Combining LLM Embeddings with Tabular Features in a Unified Scikit-learn Pipeline

We need to reformat the article into clean Markdown with paragraphs, headings, lists, code blocks, preserving facts. No extra commentary. Output only markdown.

We must break into short paragraphs, add headings where topic shifts. Use bullet/numbered lists for enumerations. Use inline code or fenced code blocks for code/commands/identifiers.

We need to preserve every fact, name, number, quotation exactly. No invention.

We must not repeat article title as heading. Title is given but we should not wrap whole answer in code fence, and not repeat title as heading.

We need to output only reformatted article body as Markdown.

Let's parse article.

Original article text includes repeated first sentence? It starts: "In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with..." then repeats same sentence again. Probably a duplication error but we must preserve exactly as given? The instruction: "Preserve every fact, name, number, and quotation exactly as given. Never invent, add, remove, or reorder information." So we must keep the duplicated sentence? It appears twice in a row. That is part of the raw article. We must keep it exactly. However we can restructure paragraphs, but cannot remove or reorder information. Reordering is not allowed. So we must keep the sequence of sentences as they appear. However we can break into paragraphs but must preserve order. Duplicated sentence will appear twice consecutively.

We need to add headings where topic clearly shifts. We need to decide headings based on content shifts: Introduction, Prerequisites, Step-by-Step Guide, etc. The article already has sections like "Introduction", "Prerequisites", "Step-by-Step Guide", "Results:", "Conclusion". Those are likely headings. We should turn those into Markdown headings (##). Also maybe "Example output:" etc.

We must not repeat article title as heading. So we won't add a heading for title.

We need to use bullet or numbered lists for enumerations. The article includes bullet points after "Topics we will cover include:" with dash items. We'll convert to markdown bullet list.

Also there is a table for pip install command: a markdown-like table with columns. We need to preserve exactly. It appears as:

| 1 | !pip install -q sentence-transformers scikit-learn pandas numpy | Remove the ! if you are working in your own Python IDE rather than a cloud notebook environment like Google Colab.

We need to keep that exactly. It's a table with two columns? Actually three columns: first column "1", second column the command, third column the note. We'll keep as is.

Also there are code blocks with line numbers like:

| 1 2 3 4 ... | import pandas as pd ...

These appear as a table with first column line numbers range, second column code lines. We need to preserve exactly. They are currently formatted as a markdown table with pipe separators. We'll keep them as code fences? The instruction: Use inline code or fenced code blocks where code, commands, or identifiers appear. So we should convert those table code blocks into fenced code blocks, preserving the code exactly (including line numbers? The line numbers are part of the table, not code. The original shows line numbers in first column. If we convert to fenced code block, we should include the code without the line numbers? But we must preserve every fact exactly. The line numbers are part of the presentation, not code. However they are not code; they are part of the table. The instruction says use fenced code blocks where code, commands, or identifiers appear. The code is the lines after the pipe. The line numbers are not code but part of the table formatting. If we change to fenced code block, we would be removing the line numbers column and the pipe formatting, which would be altering the representation. Is that allowed? We must preserve every fact, name, number, and quotation exactly as given. The line numbers are numbers; they are facts. Changing them would be removing information. So we must keep them exactly as given. However we could keep them inside a fenced code block but still include the line numbers? The original representation is a table with two columns: first column line numbers (like "1 2 3 ...") and second column the code lines. If we put the whole table inside a fenced code block, we would preserve the table exactly, but then it's not a code block per se, but it's still a fenced block showing the table. That would preserve the exact text. The instruction says use fenced code blocks where code, commands, or identifiers appear. It doesn't forbid also including other text inside. So we can wrap the table in a fenced code block to preserve formatting. However the original already uses pipe table syntax; converting to fenced block would change the markup but preserve the content exactly (including pipes). That's acceptable because we are not removing or altering any characters; we just add triple backticks before and after. That is adding markup, but not altering the content. The instruction says "Use bullet or numbered lists for enumerations, and inline code or fenced code blocks where code, commands, or identifiers appear." It doesn't say we cannot keep the original table syntax; we could keep as is. But to follow instruction, we should convert code sections to fenced code blocks. However the code sections are currently inside a table; we could extract the code lines and put them in a fenced block, but then we would lose the line numbers column. That would be removing information (the line numbers). So better to keep the table as is, but maybe we can still consider it as a fenced code block by wrapping the whole table in triple backticks, preserving the pipes and line numbers. That would add backticks but not change internal content. That seems okay.

Similarly for other code blocks like the TextEmbedder class, the pipeline assembly, etc. Those are also presented as tables with line numbers. We'll wrap each in fenced code blocks.

Also there is a snippet: "Example output:" followed by a table of classification report. We'll wrap that in fenced code block.

We need to ensure we preserve blank lines between paragraphs.

Let's outline sections:

• Opening duplicated sentence(s). Actually article begins: "In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with..." then immediately repeats same sentence again. We'll keep as two sentences maybe separated by space? We'll keep exactly as they appear: "In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with... In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks." Wait the second sentence continues: "with structured tabular features for classification tasks." The first sentence truncated with "...", but we must keep exactly as given. The raw article:

"In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with...

In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks. Topics we will cover include: - How to generate text embeddings using Hugging Face’s sentence-transformers library and wrap them in a custom scikit-learn transformer class. - How to use a ColumnTransformer to run parallel preprocessing branches for text, numeric, and categorical features simultaneously. - How to assemble and evaluate a complete, deployment-ready classification pipeline on a mixed dataset combining real text data with synthetic tabular features."

Thus we need to keep that exactly. So first line ends with "with..." then newline then second sentence starts. We'll preserve line break? The raw text shows a line break after the ellipsis? It appears as a line break. We'll keep as is.

After that, we have "Introduction". We'll make a heading ## Introduction.

Then the paragraph(s) under Introduction.

Then "Prerequisites" heading.

Then the paragraph and the pip install table.

Then "Step-by-Step Guide" heading.

Then paragraphs, then the dataset generation code table, then "Example output:" maybe as a heading? Actually it's inline text: "Example output:" then the code table. We'll keep as is.

Then "Next step is key..." paragraph.

Then the custom text transformer code table.

Then paragraph about notice.

Then paragraph about next step.

Then the pipeline assembly code table.

Then paragraph about training.

Then the training/evaluation code table.

Then "Results:" maybe heading? We'll make ## Results.

Then the classification report table.

Then paragraph about results.

Then "Conclusion" heading.

Then paragraph.

Then "No comments yet." maybe as a paragraph.

We need to ensure we don't add extra content.

Now we need to produce markdown.

We'll need to preserve blank lines between paragraphs.

Let's construct.

We'll start with the duplicated sentences exactly as given, then a blank line, then ## Introduction, then paragraph(s).

We need to be careful to not add extra spaces.

Let's copy the exact text from the article.

I'll extract the article text from the prompt.

Article:

"In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with...

In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks. Topics we will cover include: - How to generate text embeddings using Hugging Face’s sentence-transformers library and wrap them in a custom scikit-learn transformer class. - How to use a ColumnTransformer to run parallel preprocessing branches for text, numeric, and categorical features simultaneously. - How to assemble and evaluate a complete, deployment-ready classification pipeline on a mixed dataset combining real text data with synthetic tabular features. Introduction Real-world tasks like ticket triage or customer churn prediction are typically addressed by building classification models. Yet, in an increasingly data-pervaded era, the data used to construct these models and perform inference on them rarely comes in a single flavor. We are often faced with a mix of tabular, structured data of numeric and qualitative nature, as well as unstructured data like text - for instance, ticket descriptions or customer messages. Feeding these data types together into machine learning models requires effective and unified pipelines that accommodate the latest data nuances and techniques to handle them. This article shows you how to build a clean, deployment-ready solution that encapsulates embeddings generated by open-source LLMs (language models) into a unified scikit-learn pipeline, bringing together text representations and tabular features of distinct types - all based on the use of a ColumnTransformer . To illustrate its use, we will consider a classification scenario for detecting spammer users in a customer base. Prerequisites Instead of resorting to a paid API like OpenAI’s or Google Gemini’s, or a massive open-source LLM like LLaMA 3, we will use a more lightweight, CPU-friendly solution to generate embeddings from a collection of texts: Hugging Face’s sentence-transformers . Depending on your running environment, all you may need is to install the following libraries and dependencies: | 1 | !pip install -q sentence-transformers scikit-learn pandas numpy | Remove the ! if you are working in your own Python IDE rather than a cloud notebook environment like Google Colab. Step-by-Step Guide Here’s what our intended, unified scikit-learn pipeline architecture looks like: But first, we need a mixed dataset that looks reasonably realistic. For this, we adopt a hybrid approach: we pull a real dataset available on GitHub - the well-known SMS Spam Collection dataset containing users’ text messages labeled as spam or not - and augment it with synthetic tabular data features. Put together, the data will serve us to set up a customer churn/triage scenario. The code excerpt required for data generation is a bit large, but there are plenty of comments to help you understand every decision behind the synthetic data creation process: | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | import pandas as pd import numpy as np # 1. Loading base text dataset from GitHub url = "https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv" df = pd.read_csv(url, sep='\t', header=None, names=['label', 'message']) # 2. Encoding original target variable first (0 for normal/ham, 1 for spam) df['target'] = df['label'].map({'ham': 0, 'spam': 1}) # 3. Synthesising meaningful tabular features WITH realistic overlap (noise) # Without noise and some degree of overlap, the classifier we will build would # easily achieve perfection: something not quite realistic in practice. np.random.seed(42) # Account Age: Normal users can be brand new, and spammers sometimes use older hacked accounts df['account_age_days'] = np.where( df['target'] == 1, np.random.randint(1, 365, df.shape[0]), # Spam: 1 to 365 days np.random.randint(1, 1500, df.shape[0]) # Ham: 1 to 1500 days (Massive overlap) ) # Premium Status: Adding a bit more noise here df['is_premium'] = np.where( df['target'] == 1, np.random.choice(['no', 'yes'], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free np.random.choice(['no', 'yes'], df.shape[0], p=[0.80, 0.20]) # Ham: 80% free, 20% premium ) # Priority Score: Overlapping distributions so the model can't rely on this feature alone to classify customers df['priority_score'] = np.where( df['target'] == 1, np.random.uniform(0.4, 1.0, df.shape[0]), # Spam: 0.4 to 1.0 np.random.uniform(0.0, 0.7, df.shape[0]) # Ham: 0.0 to 0.7 (Overlap between 0.4 and 0.7) ) # Viewing a sample of the logically cohesive mixed data df.head(3) | Example output: The next step is key, as this is where we create the custom text transformer - see the leftmost branch in the previous diagram. In scikit-learn, this is done by creating a custom class that inherits from TransformerMixin and BaseEstimator . The requirement is to define fit() and transform() methods, just like any pre-existing data transformation class in the library (e.g. standard scalers and one-hot encoders). | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | from sklearn.base import BaseEstimator, TransformerMixin from sentence_transformers import SentenceTransformer class TextEmbedder(BaseEstimator, TransformerMixin): def init(self, model_name='all-MiniLM-L6-v2'): self.model_name = model_name self.model = None def fit(self, X, y=None): # Initializing the model in fit() to comply with sklearn cloning rules if self.model is None: self.model = SentenceTransformer(self.model_name) return self def transform(self, X, y=None): # Handling pandas DataFrame (extract the first column as a list of strings) if isinstance(X, pd.DataFrame): texts = X.iloc[:, 0].astype(str).tolist() else: texts = pd.Series(X).astype(str).tolist() # Using the specified LLM, generate and return embeddings as a 2D numpy array return self.model.encode(texts, show_progress_bar=False) | Notice that we specify the Hugging Face sentence-transformer model to use - namely all-MiniLM-L6-v2 - in the constructor method, and call the model in transform() to map texts into embeddings. Next, once we have our embeddings, we apply the parallel data preprocessing required by the other features. Since this relies entirely on already-implemented classes in scikit-learn, we can directly assemble all the type-specific preprocessing steps into an overarching, unified pipeline. We distinguish numerical columns from categorical ones, applying standard scaling to the former and one-hot encoding to the latter. Together with the previously implemented text embedding step, this gives us three processing branches that run in parallel. The way to implement this is through a ColumnTransformer object that contains a list of three “processing branches.” This mechanism keeps the whole dataset together, without the need to manually split and re-unify features. After that, we add the final stage: a random forest classifier. The entire process looks as follows: | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import classification_report # Split data X = df[['message', 'account_age_days', 'priority_score', 'is_premium']] y = df['target'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Define column groups text_features = ['message'] numeric_features = ['account_age_days', 'priority_score'] categorical_features = ['is_premium'] # Build the ColumnTransformer preprocessor = ColumnTransformer( transformers=[ ('text', TextEmbedder(), text_features), ('num', StandardScaler(), numeric_features), ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features) ], remainder='drop' # Drop any columns not explicitly defined ) # Assemble the final pipeline pipeline = Pipeline(steps=[ ('preprocessor', preprocessor), ('classifier', RandomForestClassifier(n_estimators=100, random_state=42)) ]) | Now that we have assembled the entire pipeline, it’s time to try it out! The final piece of code trains the model - a process that, thanks to the pipeline encapsulation, implicitly carries out all the preceding data preparations - and evaluates it on the test set we set aside earlier: | 1 2 3 4 5 6 7 8 | # Training the model (this will take a moment to download the HF model and embed the texts) print("Training pipeline...") pipeline.fit(X_train, y_train) # Evaluating on test examples print("Predicting and evaluating...") y_pred = pipeline.predict(X_test) print(classification_report(y_test, y_pred)) | Results: | 1 2 3 4 5 6 7 8 9 | Predicting and evaluating... precision recall f1-score support 0 0.99 1.00 0.99 966 1 1.00 0.91 0.95 149 accuracy 0.99 1115 macro avg 0.99 0.95 0.97 1115 weighted avg 0.99 0.99 0.99 1115 | These results are pretty decent. Part of the reason is that the real dataset used for the labeled texts is known for being easily class-separable and therefore not hard to classify with high accuracy. We also intentionally added noise and overlap when creating the other synthetic attributes to introduce a bit of challenge for our classifier - otherwise, it might have achieved 100% accuracy, which would not be very informative. Conclusion This article tackled an increasingly common problem in the AI and data science landscape: leveraging text data and combining it with structured data features traditionally fed to downstream machine learning models for predictive tasks like classification. We used scikit-learn’s transformer classes and a pre-trained language model to build a unified pipeline that cleanly and elegantly processes these mixed data types, yielding a robust and easily reusable solution. No comments yet."

We need to preserve exactly.

Now we need to add headings where topic shifts. The article already has words "Introduction", "Prerequisites", "Step-by-Step Guide", "Results:", "Conclusion". We'll convert those to headings.

We must not repeat article title as heading.

We need to ensure we don't add extra headings besides those.

We'll keep the text as is, but we will insert markdown heading syntax before those words, and ensure they are on their own line.

We also need to break into short paragraphs separated by blank line. We'll need to parse where paragraphs naturally break: after each sentence or after certain cues. We'll just follow the existing line breaks in the raw text? The raw text appears as a continuous block with no line breaks except maybe after some sentences. But we can insert blank lines to separate paragraphs logically.

We must not reorder information. Adding blank lines is allowed as restructuring (breaking into paragraphs). That's fine.

We need to preserve every fact exactly; adding blank lines does not change content.

Thus we will produce markdown with headings and paragraphs.

Let's construct.

Start with the duplicated sentences.

We'll keep them exactly as they appear, including the line break after the ellipsis? The raw shows:

"In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with...

In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks."

We'll keep that exactly, then a blank line.

Then ## Introduction

Then the paragraph(s) under Introduction: from "Real-world tasks like ticket triage..." up to before "Prerequisites". Let's copy that text.

Introduction paragraph:

"Real-world tasks like ticket triage or customer churn prediction are typically addressed by building classification models. Yet, in an increasingly data-pervaded era, the data used to construct these models and perform inference on them rarely comes in a single flavor. We are often faced with a mix of tabular, structured data of numeric and qualitative nature, as well as unstructured data like text - for instance, ticket descriptions or customer messages. Feeding these data types together into machine learning models requires effective and unified pipelines that accommodate the latest data nuances and techniques to handle them. This article shows you how to build a clean, deployment-ready solution that encapsulates embeddings generated by open-source LLMs (language models) into a unified scikit-learn pipeline, bringing together text representations and tabular features of distinct types - all based on the use of a ColumnTransformer . To illustrate its use, we will consider a classification scenario for detecting spammer users in a customer base."

We need to preserve exactly, including spaces and punctuation. Note there is a space before the period after "ColumnTransformer ." Actually there is a space before the period: "ColumnTransformer .". We'll keep.

After that, blank line, then ## Prerequisites.

Then the Prerequisites paragraph: "Instead of resorting to a paid API like OpenAI’s or Google Gemini’s, or a massive open-source LLM like LLaMA 3, we will use a more lightweight, CPU-friendly solution to generate embeddings from a collection of texts: Hugging Face’s sentence-transformers . Depending on your running environment, all you may need is to install the following libraries and dependencies:"

Then blank line, then the pip install table.

We need to keep the table exactly as given:

| 1 | !pip install -q sentence-transformers scikit-learn pandas numpy | Remove the ! if you are working in your own Python IDE rather than a cloud notebook environment like Google Colab.

We'll keep that.

After that, blank line, then ## Step-by-Step Guide.

Then the paragraph: "Here’s what our intended, unified scikit-learn pipeline architecture looks like: But first, we need a mixed dataset that looks reasonably realistic. For this, we adopt a hybrid approach: we pull a real dataset available on GitHub - the well-known SMS Spam Collection dataset containing users’ text messages labeled as spam or not - and augment it with synthetic tabular data features. Put together, the data will serve us to set up a customer churn/triage scenario. The code excerpt required for data generation is a bit large, but there are plenty of comments to help you understand every decision behind the synthetic data creation process:"

Note there is no period after "looks like:"? Actually there is a colon then space then "But". We'll keep.

Then blank line, then the dataset generation code table.

We need to keep that table exactly:

| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | import pandas as pd import numpy as np # 1. Loading base text dataset from GitHub url = "https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv" df = pd.read_csv(url, sep='\t', header=None, names=['label', 'message']) # 2. Encoding original target variable first (0 for normal/ham, 1 for spam) df['target'] = df['label'].map({'ham': 0, 'spam': 1}) # 3. Synthesising meaningful tabular features WITH realistic overlap (noise) # Without noise and some degree of overlap, the classifier we will build would # easily achieve perfection: something not quite realistic in practice. np.random.seed(42) # Account Age: Normal users can be brand new, and spammers sometimes use older hacked accounts df['account_age_days'] = np.where( df['target'] == 1, np.random.randint(1, 365, df.shape[0]), # Spam: 1 to 365 days np.random.randint(1, 1500, df.shape[0]) # Ham: 1 to 1500 days (Massive overlap) ) # Premium Status: Adding a bit more noise here df['is_premium'] = np.where( df['target'] == 1, np.random.choice(['no', 'yes'], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free np.random.choice(['no', 'yes'], df.shape[0], p=[0.80, 0.20]) # Ham: 80% free, 20% premium ) # Priority Score: Overlapping distributions so the