Where exactly is the card in this photo? Image segmentation model inside a maxed-out lambda container

A crooked phone photo goes in, a clean straight card image comes out - no GPU, and nothing running when nobody uploads. It runs on the biggest lambda AWS sells, and the biggest is not the same as enough. Introduction This is the 3rd article in the series about the image processing pipeline. In two previous articles we went over the whole image processing pipeline, which process AWS Builder Cards images: From anonymous photo to a published page: An event-driven, AI, image processing pipeline on AWS and also over image classification model running in lambda container, which filters out the uploaded images and filter only valid AWS Builder Card images: Is this even a valid card? Zero-shot image classification model in a lambda container Today's article is about vision segmentation model, which also runs in a lambda container, but it's role is to process the filtered image - remove background, straighten it and crop it. Specifically about this part: The problem Somebody photographs an AWS Builder Card lying on their desk and uploads it to my collection. What arrives is not a card - it is a "photo" of a card. Tilted and with some background around it. One lambda has already looked at it before - a small, container based lambda card-detect , which filtered it as valid AWS Builder Card. See this article for more info. Now we have to clean that picture (straighten, crop, remove background), which is a job of lambda function image-processor . That lambda answers the question: Where exactly is the card in this photo? The coworking between the two lambdas is simple: after the image is put into S3 images/raw/ and filtered by card-detect as a valid one, image-processor overtakes and start to processing the image. Can I run it in the container? To process the image as describe above, I need an image segmentation model. The one that goes pixel by pixel and says: "card, background, card, background..." I need it to run on CPU , inside a lambda container from the same reasons as card-detect (inside AWS, cheap, serverless). To find a right model to do the job I need to know the answers to the same questions, as with card-detect lambda running CLIP ViT-B-32, laion2b_s34b_b79k : - How much CPU and memory does it need? - Where do the weights come from? - What runs at the inference? The inference In this case, this is the easiest one to answer. It runs on onnxruntime behind a small library called rembg , which means no PyTorch and no training framework anywhere in the image. The other two get a section each further down, and question 1 is where the story is: the model I picked first barely fit at 10 GB . Lambda also ties vCPU to memory, so that number decides how fast this thing runs, not just whether it survives. So this is an article about memory, about the bill, and about the geometry that happens after the model has stopped talking. One thing before we start, so it does not ambush you later: there are two models in this lambda. BiRefNet finds the card. Right at the end, a Bedrock vision model reads the text off the finished image. But let's start from the beginning from rembg import new_session SESSION = new_session("birefnet-general-lite") The model As an image segmentation model, I decided to go with BiRefNet-General-Lite , loaded through rembg . The Lite is also the answer to the memory question. The memory story Remember it works on my machine meme from previous article? Here is where I really earned it. What I was working with locally was the BiRefNet-General model. It worked fine (on my machine 🤣), so I shipped it. What I never did was to watch how much memory it was eating while it worked. I started the container at 4 GB memory. The first couple of invocations showed me the problem: Runtime.OutOfMemory, Max Memory Used: 4095 MB of 4096 . So I went to maximum I could - 10 GB memory. It worked fine, but not as per CloudWatch : Max Memory Used: 9930 MB . That's wonderful 97% of maximum container's capacity. So yet it works, but then I tested 125 MB image and it crashed. There is no 12 GB to escape to, no bigger instance type, no flag to ask for more. With standard 1: largest = 1 + int(np.argmax(stats[1:, cv2.CC_STAT_AREA])) mask = np.where(labels == largest, 255, 0).astype(np.uint8) return mask The returned mask for Golden Jacket card looks like this: And this is where the segmentation model's job ends. It has separated the card from the background and everything that follows is a deterministic math. Fight the rounded corners When I was thinking about straightening the card, I also thought about the eye-to-brain path. Taking a photo of a card from an angle, turns its rectangular shape into a quadrilateral, as you can see in the mask above. How do I perform a perspective rectification and turn it back into a rectangle? The answer is in the corners. Lambda needs to identify the card's four corners first. A standard approach would be use OpenCV function approxPolyDP , which simplifies the detected outline into four corner points. But here's where Builder Cards fight back: their corners are rounded, so the contour does not have four sharp corners. For this reason approxPolyDP sometimes mistakenly placed a corner somewhere inside the rounded section, like here: So instead calling a approxPolyDP , the more logical step would be if lambda fits a straight line along each of the four sides of the card. Now the mask comes handy, because it tells exactly where the card's contour is. Before fitting the lines, it temporarily rotate the contour points so the card is approximately upright. This lets it separate the contour points into top, bottom, left and right edges. Because the rounded corners would distort the straight line, lambda excludes the outer 18% at each end, when selecting the contour points used for each edge. In other words, it uses the middle part of each side, where the contour follows the card's straight edge. It then fits a straight line to each of those four sections. Where two neighbouring lines intersect gives us a corner. Because the lines represent the card's straight edges rather than its rounded physical corners, those intersections can lie slightly outside the visible card. Those four intersections become the card's four corner points. Lambda then transforms those coordinates back to the original image coordinate system. The card itself is not straightened yet. These points are passed to the next step, where they are used for the actual perspective rectification. def find_quad(mask): # ... (docstring and contour extraction omitted) ... # Rotate the contour so the card becomes approximately axis-aligned. (cx, cy), (rw, rh), ang = cv2.minAreaRect(c) M = cv2.getRotationMatrix2D((cx, cy), ang, 1.0) P = _rotate_pts(pts, M) # Compute the bounding box of the rotated contour. minx, maxx = P[:, 0].min(), P[:, 0].max() miny, maxy = P[:, 1].min(), P[:, 1].max() W, H = maxx - minx, maxy - miny bx, by = 0.18 * W, 0.18 * H # Ignore the rounded corners when selecting edge points. cxlo, cxhi = minx + bx, maxx - bx cylo, cyhi = miny + by, maxy - by # Split contour points into the four card edges. top = P[(P[:, 1] cxlo) & (P[:, 0] maxy - by) & (P[:, 0] > cxlo) & (P[:, 0] = 15: lt, lb = _fit_line(top), _fit_line(bottom) ll, lr = _fit_line(left), _fit_line(right) tl, tr = _intersect(lt, ll), _intersect(lt, lr) br, bl = _intersect(lb, lr), _intersect(lb, ll) # ... (rotate corners back and fallback methods omitted) ... But wait, should I blindly trust four invisible points? Well, No. For debugging reasons, encode_corners() draws the detected quadrilateral and its four corner points onto the original image., which is saved as images/raw/ __corners.png . In case a card comes out cropped wrong, I open this overlay and see immediately whether the corners were the problem. Straighten it and crop it Now that we have the four corners identified, OpenCV calls getPerspectiveTransform to calculate a perspective transformation that maps those four points onto the four corners of a rectangle, and then warpPerspective applies that transformation to the original image. This is where the card is actually straightened and warped. The result is a rectangular image containing the straightened card, but the background is still present around the rounded corners. This is where the mask comes back in. It goes through the exact same perspective transformation as the photo. Once it's straight and perfectly aligned into a rectangle, lambda can replace those corner pixels with the configured background color. def rectify(img_bgr, quad, mask, background="white"): # ... (docstring omitted) ... # Order the detected corners consistently. src = order_corners(quad) tl, tr, br, bl = src # Compute the dimensions of the output rectangle. w = int(max( np.linalg.norm(br - bl), np.linalg.norm(tr - tl), )) h = int(max( np.linalg.norm(tr - br), np.linalg.norm(tl - bl), )) # ... (invalid-size guard omitted) ... # Map the four card corners to the four corners of the rectangle. dst = np.float32([ [0, 0], [w - 1, 0], [w - 1, h - 1], [0, h - 1], ]) M = cv2.getPerspectiveTransform(src, dst) # Straighten the original image. warped = cv2.warpPerspective(img_bgr, M, (w, h)) # Straighten the mask using exactly the same transformation. warped_mask = cv2.warpPerspective( mask, M, (w, h), flags=cv2.INTER_NEAREST, ) # Threshold and erode the transformed mask. _, warped_mask = cv2.threshold( warped_mask, 127, 255, cv2.THRESH_BINARY ) er = cv2.getStructuringElement( cv2.MORPH_ELLIPSE, (3, 3) ) warped_mask = cv2.erode( warped_mask, er, iterations=1 ) # ... (transparent-background branch omitted) ... # Replace the background with the selected solid color FILLS = {"white": 255, "black": 0} if background not in FILLS: logger.warning( "[ImageProcessor] unknown BACKGROUND=%r (expected white, black or none) - falling back to black", background, ) fill = FILLS.get(background, 0) outside = warped_mask == 0 warped[outside] = fill return warped In this step, we still don't have a proper png as a r