Step 3: Cleaning Data

By now you are already proficient at building models — doesn't it feel simple? But the data you get is often not necessarily "clean" — it contains sensor spikes, data entry errors, and genuine process anomalies. Next, let's pick out these "impurities".

There are three cleaning paths, which can be used in combination:

Path	Who it suits	Entry point
Manual lasso	You trust your own eyes and want to confirm point by point	Toolbar → Lasso → Delete Data
Algorithm detection	You want the algorithm to give a candidate list first	The anomaly detection button inside the chart
AI automatic cleaning	You want a diagnosis report in one step	AI Analysis → AI Cleaning

---

🎯 Path 1: Circle Anomalies Like Playing a Game (Lasso)

If you trust your own eyes more, try our lasso tool.

1. Open some charts in the model views, such as the T1-T2 relationship plot.
2. See those red dots far away from the rest outside the red circle? They are very likely outliers!
3. Hold down the left mouse button and draw a circle around them. ⭕️
4. Release! Those points have been "captured" and turn into the red selected state.

🔴🔵 Red and Blue Groups: Do a Comparative Analysis Along the Way

When you circle a second group of data, those points are marked blue. The red and blue groups are the foundation of the platform's "Comparative Analysis" feature:

• Manual: use the lasso to circle the red group and the blue group separately
• Shortcut: in a chart or table, use Shift + click to select the red group and Ctrl + click to select the blue group

Once both groups are selected, click Comparative Analysis in the toolbar, and the system will automatically generate a new chart containing difference analysis and contribution metrics.

💡 Going one step further: use AI Comparative Analysis to let the large model directly interpret exactly how these two groups differ and what might be causing it. See AI Capabilities Overview for details.

---

🔍 Path 2: Let the Algorithm Provide a Candidate List First

Besides circling by eye, the platform has two built-in unsupervised anomaly detection algorithms, which can pick out suspicious samples first for you to confirm.

Algorithm	Location	Characteristics
Isolation Forest	Scatter plots such as t1-t2 / loading plot	Isolates samples through random partitioning; sensitive to global outliers
LOF (Local Outlier Factor)	Prediction scatter plot	Based on local density; more sensitive to samples in locally sparse regions

How to use it: Click the corresponding button in the toolbar above the chart; once detection is complete, a message appears:

• Detected N outliers —— these points are highlighted
• No obvious outliers detected —— the current chart data distribution is normal

💡 It is normal for the two algorithms to disagree: Isolation Forest focuses on "being far off overall", while LOF focuses on "being lonely within its own neighborhood". Samples flagged by both are usually the ones most worth investigating first.

---

🤖 Path 3: Hand It Over to AI Automatic Cleaning

Entry: Toolbar AI Analysis → AI Cleaning

AI automatically completes three things in sequence:

1. Analyze model metrics —— read R²X, R²Y, Q²Y, and the number of components
2. Generate multi-dimensional charts —— decide on its own which charts to draw to support the analysis
3. Diagnose abnormal data —— locate out-of-limit samples, automatically highlight and circle them in red, and give possible business-level causes one by one

The resulting report includes a list of abnormal samples, root cause speculation, and cleaning suggestions, and can be exported to Markdown or Word for retention.

⚠️ AI is only responsible for "circling them out" and "giving suggestions"; it will not automatically delete any data — the final decision on whether to remove them is yours.

See AI Capabilities Overview for details.

---

🗑️ Batch Deletion

After circling (or detecting), click the Delete Data button in the toolbar.

• A dialog box pops up; click Show Selected Only to list all the "bad guys" you just circled.
• Once confirmed, click "Delete".

💥 Bang! The abnormal data is gone!

💡 A delete operation creates a new filtered model; the original model is still kept in the model list on the right and is not destroyed — you can switch back at any time to compare.

---

🔄 After Deleting: Retrain

After deleting abnormal samples, follow these steps to bring the metrics back up:

1. Rename the newly generated model (to make it easy to tell "before cleaning" from "after cleaning")
2. Click to activate it in the model list on the right (the status switches to Active)
3. Click Fit to retrain

Then compare the metrics of the two models:

Observation point	Description
Did Q²Y improve?	Improved → what you removed was real noise; decreased → you may have deleted useful information
Did the gap between R²Y and Q²Y narrow?	Narrowed → overfitting has been alleviated
Is the score plot more clustered?	Yes → the data structure is clearer

💡 Cleaning is an iterative process: delete data → Fit again → look at the charts again → delete more data. Compare Q²Y in every round until the metrics no longer improve noticeably. Be careful not to "delete data to make things look good" — removing real but uncommon operating conditions will make the model fail once it goes live.

---

⚖️ How to Choose Among the Three Paths

Scenario	Recommended path
Few outliers, obvious locations	Manual lasso
Large data volume, too many to find by eye	Algorithm detection (Isolation Forest / LOF)
Unfamiliar with the data, need business interpretation	AI automatic cleaning
Need to explain to others "why these were removed"	AI automatic cleaning (reports can be retained and exported)

---

🔗 Related Reading

• AI Capabilities Overview —— Full explanation of AI Cleaning and AI Comparative Analysis
• Model Exploration and Parameter Optimization —— Do in-depth analysis after cleaning is complete
• Step 4: Result Output —— Export the clean data and reports