The region between the circle and the square can be expressed by an angle and a distance from the circle point at that angle to the square point at that angle.
In this image, you can see the resulting shape of the above approach. The $\color{red}{\text{$u$ coordinate is expressed in red}}$, and the $\color{green}{\text{$v$ coordinate is expressed in green}}$. The $u$ coordinate begins at $u=0$, forming a circle, and ends at $u=1$, forming a square.
To achieve the above result starting from cartesian coordinates, first convert the cartesian coordinates to polar coordinates so that you have an angle measurement spanning around the shape and a radius measurement spanning the thickness of the shape. Second, scale the thickness measurement to match the difference in distance between the center of the shapes and their edges at any given angle.
To convert the $(x,y)$ cartesian coordinates to $(r,\theta)$ polar coordinates:
$$r=f_1(x,y)=\sqrt{|x^2+y^2|}$$
$$\theta=f_2(x,y)=
\begin{cases}
\frac{\pi}{2}-\arctan(\frac{x}{y}), & \text{if $y>=0$} \\
\frac{3\pi}{2}-\arctan(\frac{x}{y}), & \text{if $y>0$}
\end{cases}$$
To convert the $(r,\theta)$ polar coordinates to $(u,v)$ region coordinates in the space between the circle of diameter $a$ and square of side length $b$:
$$u=f_3(r,\theta)=
\begin{cases}
(r-\frac{a}{2})(b\sqrt{|1+(\tan\theta)^2|}-\frac{a}{2}), & \text{if $|\cos\theta|>=|\sin\theta|$ or more simply $|x|>=|y|$ if $x$ and $y$ are handy} \\
(r-\frac{a}{2})(b\sqrt{|1+(\frac{1}{\tan\theta})^2|}-\frac{a}{2}), & \text{if $|\cos\theta|<|\sin\theta|$ or more simply $|x|<|y|$ if $x$ and $y$ are handy}
\end{cases}$$
$$v=f_4(\theta)=\frac{\theta}{2\pi}$$
The conversion above can be simplified by substituting $c$ for the $\theta$ expressions:
$$c=g_3(\theta)=
\begin{cases}
\tan\theta, & \text{if $|\cos\theta|>=|\sin\theta|$ or more simply $|x|>=|y|$ if $x$ and $y$ are handy} \\
\frac{1}{\tan\theta}, & \text{if $|\cos\theta|<|\sin\theta|$ or more simply $|x|<|y|$ if $x$ and $y$ are handy}
\end{cases}$$
$$u=g_4(r,c)=(r-\frac{a}{2})(b\sqrt{|1+c^2|}-\frac{a}{2})$$
$$v=g_5(\theta)=\frac{\theta}{2\pi}$$
You can see the equation in action with this shader here:
#version 300 es
precision highp float;
precision highp sampler2D;
// normalized coordinates, (0,0) is the bottom left
in vec2 uv;
// resulting fragment color, you may name it whatever you like
out vec4 out_color;
// size of the canvas in pixels
uniform vec2 u_resolution;
// elapsed time since shader compile in seconds
uniform float u_time;
// mouse pixel coordinates are in canvas space, (0,0) is top left
uniform vec4 u_mouse;
// texture array
uniform sampler2D u_textures[16];
const float pi = 3.14159265;
const float pi2 = pi * 2.0;
// Converts cartesian to radians in range [0, pi2]
float atan2positive(vec2 v) {
// Shift range to [pi, 0] repeated across x axis
float a = atan(v.x / v.y) + pi / 2.0;
// Flip regions and shift lower region past upper to final single region
return v.y >= 0.0 ? pi - a : pi2 - a;
}
// Converts cartesian unit square to polar unit circle
vec2 cartesianToNormalizedPolar(vec2 cartesian) {
return vec2(atan2positive(cartesian)/pi2, length(cartesian));
}
void main(){
// Find bounding aspect
float minResolution = min(u_resolution.x, u_resolution.y);
// Find the normalized margin size
vec2 centeringShift = (u_resolution - minResolution) / minResolution;
// Cartesian coordinates in ranges <x[-1, 1], y[-1, 1]>
vec2 cartesian = 2.0 * uv * u_resolution / minResolution - 1.0 - centeringShift;
// Polar coordinates in ranges <angle[0, 1], radius[0, 1]>
vec2 polar = cartesianToNormalizedPolar(cartesian);
// Radius of the circle to act as the zero coordinate
float a = 0.0;
// Length of one side of the square to act as the one coordinate
float b = 1.0;
// Distance to the square when on the left or right side
float sx = length(vec2(b, b * sin(polar.x*pi2)));
// Distance to the square when on the top or bottom side
float sy = length(vec2(b * cos(polar.x*pi2), b));
// Distance to the square when on any side
float s = abs(cartesian.x) >= abs(cartesian.y) ? sx : sy;
// Ratio of the distance from the circle to the square the pixel has traveled
float d = (polar.y - a) / (s - a);
// Inscription coordinates in the region between the circle and square
vec2 inscription = vec2(polar.x, pow(d, 15.0));
out_color = vec4(inscription, 0, 1);
}
