Data converter Widget by rabii-chaarani · Pull Request #76 · Loop3D/plugin_loopstructural

rabii-chaarani · 2026-01-27T23:27:32Z

No description provided.

…tool

Copilot

Pull request overview

This pull request introduces a comprehensive "Data Converter Widget" that enables users to convert geological datasets for use within LoopStructural through both automatic and manual conversion workflows.

Changes:

Added a new data conversion widget with automatic and manual conversion modes
Implemented utility functions to convert GeoPandas GeoDataFrames and pandas DataFrames to QGIS vector layers
Integrated the data conversion widget into the main Loop widget interface
Added tests for the NTGS configuration model

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
tests/unit/gui/test_ntgs_configuration_model.py	Unit tests for configuration model (imports incorrect class names)
loopstructural/main/vectorLayerWrapper.py	New functions to convert DataFrames/GeoDataFrames to QGIS layers
loopstructural/gui/loop_widget.py	Integration of data conversion widget into main interface
loopstructural/gui/data_conversion/data_conversion_widget.py	Main widget implementation with automatic and manual conversion modes
loopstructural/gui/data_conversion/configuration.py	Configuration state management for NTGS data types
loopstructural/gui/data_conversion/init.py	Package initialization exposing DataConversionWidget

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-27T23:46:25Z

tests/unit/gui/test_ntgs_configuration_model.py

@@ -0,0 +1,33 @@
+import pytest
+
+from loopstructural.gui.data_conversion.configuration import NtgsConfig, NtgsConfigurationModel


The test imports NtgsConfig and NtgsConfigurationModel, but the actual configuration module defines these classes as Config and ConfigurationState. This import will fail at runtime. The imports should be updated to match the actual class names in loopstructural/gui/data_conversion/configuration.py.

Copilot · 2026-01-27T23:46:26Z

loopstructural/main/vectorLayerWrapper.py

+    for _, row in df.iterrows():
+        feat = QgsFeature(layer.fields())
+        attrs = []
+        for column in df.columns:
+            val = row[column]
+            if pd.isna(val):
+                val = None
+            elif isinstance(val, np.generic):
+                try:
+                    val = val.item()
+                except Exception:
+                    pass
+            attrs.append(val)
+        feat.setAttributes(attrs)
+        features.append(feat)


Using iterrows() is generally inefficient for large DataFrames as it creates a new Series object for each row. For better performance with large datasets, consider using itertuples() or vectorized operations where possible.

Copilot · 2026-01-27T23:46:26Z

loopstructural/gui/loop_widget.py

+        self.data_conversion_widget = DataConversionWidget(
+            self,
+            data_manager=self.data_manager,
+            project=self.data_manager.project if self.data_manager else None,


Accessing self.data_manager.project could raise an AttributeError if data_manager is not None but lacks a project attribute. Consider using getattr(self.data_manager, 'project', None) for safer attribute access.

Suggested change

project=self.data_manager.project if self.data_manager else None,

project=getattr(self.data_manager, "project", None) if self.data_manager else None,

loopstructural/gui/data_conversion/configuration.py

Copilot · 2026-01-27T23:46:26Z

loopstructural/main/vectorLayerWrapper.py

+    return layer
+
+
+def QgsLayerFromDataFrame(dataframe, layer_name: str = "Converted Table"):


The function name QgsLayerFromDataFrame uses PascalCase, which is inconsistent with existing functions in this file like qgsLayerToGeoDataFrame, qgsLayerToDataFrame, and qgsRasterToGdalDataset that use camelCase. Consider renaming to qgsLayerFromDataFrame to maintain consistency with the existing naming convention in this file.

Suggested change

def QgsLayerFromDataFrame(dataframe, layer_name: str = "Converted Table"):

def qgsLayerFromDataFrame(dataframe, layer_name: str = "Converted Table"):

Copilot · 2026-01-27T23:46:28Z

loopstructural/main/vectorLayerWrapper.py

+                try:
+                    if temp.createFromWkt(text):
+                        return temp
+                except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

loopstructural/main/vectorLayerWrapper.py

Copilot · 2026-01-27T23:46:28Z

loopstructural/main/vectorLayerWrapper.py

+            temp = QgsCoordinateReferenceSystem(crs_info)
+            if temp.isValid():
+                return temp
+        except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

except Exception:

# Ignore failures when interpreting CRS from string; fall back to default CRS.

Copilot · 2026-01-27T23:46:29Z

loopstructural/main/vectorLayerWrapper.py

+                except Exception:
+                    pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as exc:

logger.debug("Failed to convert numpy scalar %r using .item(): %s", val, exc)

Copilot · 2026-01-27T23:46:29Z

loopstructural/main/vectorLayerWrapper.py

+                except Exception:
+                    pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as exc:

# Fall back to the original NumPy scalar if conversion fails.

logger.debug("Failed to convert NumPy scalar %r with .item(): %s", val, exc)

lachlangrose · 2026-01-28T00:21:23Z

Can you make the data converter its own dialog or dock widget? I think it is an addition to the modelling process rather than a first step as you can use the modelling without using the data conversion. Also do you have a demo dataset I can use to test it?

lachlangrose · 2026-01-28T21:22:25Z

@rabii-chaarani looks good, couple of easy things

use the column matcher from main.helper to automatically guess the appropriate file from the available files.
Restrict the layer types to the appropriate geometries e.g. geology should be polygon not line or point, fault/fold should be line and structure points.
Don't add xyz to the fault layer as they are a line layer, maybe add it as centre x, centre y, centre z.

Future idea

Add a logging capabilities to report to the user what was done by the tool. E.g. column x changed to column y, create a file from a layer etc.
Use the QGIS layers directly, not the underlying shapefile as the underlying shapefile is not compatible with filters.

…tool

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 13 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-30T00:57:42Z

loopstructural/main/m2l_api.py

    if unit_name_field != 'UNITNAME' and unit_name_field in geology_gdf.columns:
        geology_gdf = geology_gdf.rename(columns={unit_name_field: 'UNITNAME'})


The unit name field is being renamed twice: once at line 492-494 and again at lines 540-541. This is redundant and could lead to confusion. The second rename should be removed as the first one already handles this transformation.

Suggested change

if unit_name_field != 'UNITNAME' and unit_name_field in geology_gdf.columns:

geology_gdf = geology_gdf.rename(columns={unit_name_field: 'UNITNAME'})

Copilot · 2026-01-30T00:57:42Z

loopstructural/main/m2l_api.py

    )
    sampled_contacts_gdf = qgsLayerToGeoDataFrame(sampled_contacts)
-    structure_gdf = qgsLayerToGeoDataFrame(structure)
+    structure_gdf = qgsLayerToGeoDataFrame(structure) 


Trailing whitespace found on line 460. This should be removed to maintain code cleanliness and consistency with the codebase style.

loopstructural/gui/data_conversion/data_conversion_widget.py

loopstructural/gui/map2loop_tools/thickness_calculator_widget.py

loopstructural/gui/data_conversion/configuration.py

Copilot · 2026-01-30T00:57:44Z

loopstructural/gui/data_conversion/data_conversion_widget.py

+            if self.project is not None:
+                try:
+                    combo.setProject(self.project)
+                except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

except Exception:

# Some QGIS/Qt environments may not support setProject or may raise here;

# failure to bind the project is non-fatal, so we intentionally ignore it.

Copilot · 2026-01-30T00:57:44Z

loopstructural/main/vectorLayerWrapper.py

+                try:
+                    data = bytes(data)
+                except Exception:
+                    pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

pass

# Best-effort conversion to bytes; if this fails, fall back to using the

# original data and let the subsequent fromWkb call handle it.

logger.debug(

"Failed to convert WKB data to bytes in _geometry_from_value",

exc_info=True,

)

Copilot · 2026-01-30T00:57:44Z

loopstructural/main/vectorLayerWrapper.py

+                except Exception:
+                    pass
+    try:
+        epsg = crs_info.to_epsg()
+        if epsg:
+            return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))
+    except Exception:
+        pass
+    if isinstance(crs_info, str):
+        try:
+            temp = QgsCoordinateReferenceSystem(crs_info)
+            if temp.isValid():
+                return temp
+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

try:

epsg = crs_info.to_epsg()

if epsg:

return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))

except Exception:

pass

if isinstance(crs_info, str):

try:

temp = QgsCoordinateReferenceSystem(crs_info)

if temp.isValid():

return temp

except Exception:

pass

except Exception as e:

logger.debug("Failed to create CRS from WKT using createFromWkt: %r (%s)", text, e)

try:

epsg = crs_info.to_epsg()

if epsg:

return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))

except Exception as e:

logger.debug("Failed to derive EPSG from CRS info %r: %s", crs_info, e)

if isinstance(crs_info, str):

try:

temp = QgsCoordinateReferenceSystem(crs_info)

if temp.isValid():

return temp

except Exception as e:

logger.debug("Failed to construct CRS from string %r: %s", crs_info, e)

Copilot · 2026-01-30T00:57:44Z

loopstructural/main/vectorLayerWrapper.py

+                except Exception:
+                    pass
+    try:
+        epsg = crs_info.to_epsg()
+        if epsg:
+            return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))
+    except Exception:
+        pass
+    if isinstance(crs_info, str):
+        try:
+            temp = QgsCoordinateReferenceSystem(crs_info)
+            if temp.isValid():
+                return temp
+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

try:

epsg = crs_info.to_epsg()

if epsg:

return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))

except Exception:

pass

if isinstance(crs_info, str):

try:

temp = QgsCoordinateReferenceSystem(crs_info)

if temp.isValid():

return temp

except Exception:

pass

except Exception as exc:

logger.debug(

"Failed to create CRS from WKT using legacy createFromWkt: %s",

exc,

)

try:

epsg = crs_info.to_epsg()

if epsg:

return QgsCoordinateReferenceSystem.fromEpsgId(int(epsg))

except Exception as exc:

logger.debug(

"Failed to obtain EPSG code from CRS info %r: %s",

crs_info,

exc,

)

if isinstance(crs_info, str):

try:

temp = QgsCoordinateReferenceSystem(crs_info)

if temp.isValid():

return temp

except Exception as exc:

logger.debug(

"Failed to construct CRS from string %r: %s",

crs_info,

exc,

)

Copilot · 2026-01-30T00:57:45Z

loopstructural/main/vectorLayerWrapper.py

+            elif isinstance(val, np.generic):
+                try:
+                    val = val.item()
+                except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

except Exception:

# If conversion to a native Python scalar fails, keep the original numpy scalar.

rabii-chaarani added 11 commits November 24, 2025 14:17

test: add unit tests for ConfigurationModel

ba73409

feat: add DataConversionWidget

f0381eb

feat: add data conversion GUI components

62807e7

feat: implement data conversion configuration helpers

a195cc1

feat: DataConversionWidget implemenation

b1e8112

fix: improve ui handling

40d2945

fix: update config file

cb9dfcb

refactor: renamed layers

1da4630

refactor: update geodataframe conversion

47ce01b

feat: add run functionality to manual conversion

076a08c

Merge remote-tracking branch 'origin/dev-0.1.12' into data_converter_…

ebc532a

…tool

lachlangrose added UI quality labels Jan 27, 2026

rabii-chaarani requested a review from lachlangrose January 27, 2026 23:27

lachlangrose requested a review from Copilot January 27, 2026 23:40

Copilot started reviewing on behalf of lachlangrose January 27, 2026 23:40 View session

Copilot AI reviewed Jan 27, 2026

View reviewed changes

rabii-chaarani and others added 2 commits January 28, 2026 09:22

fix: remove manual conversion code

c108f56

fix: add loopdataconverter to requirements

303ca20

fix: added data converter as dialog

ba3d524

rabii-chaarani added 4 commits January 29, 2026 11:38

feat: added along section thickness calc

d1afe01

refactor: added along section code

dbe347c

Merge remote-tracking branch 'origin/dev-0.1.12' into data_converter_…

33f12aa

…tool

feat: add layer guessing and filters to conversion widget

d901fe3

lachlangrose requested a review from Copilot January 30, 2026 00:51

Copilot started reviewing on behalf of lachlangrose January 30, 2026 00:52 View session

Copilot AI reviewed Jan 30, 2026

View reviewed changes

fix: applying some copilot changes

2c0fe3b

Merge branch 'dev-0.1.12' into data_converter_tool

2e59d9c

lachlangrose approved these changes Jan 30, 2026

View reviewed changes

lachlangrose merged commit ea99e14 into dev-0.1.12 Jan 30, 2026
1 check passed

lachlangrose deleted the data_converter_tool branch January 30, 2026 04:15

		@@ -0,0 +1,33 @@
		import pytest

		from loopstructural.gui.data_conversion.configuration import NtgsConfig, NtgsConfigurationModel

	project=self.data_manager.project if self.data_manager else None,
	project=getattr(self.data_manager, "project", None) if self.data_manager else None,

		return layer


		def QgsLayerFromDataFrame(dataframe, layer_name: str = "Converted Table"):

	except Exception:
	except Exception:
	# Ignore failures when interpreting CRS from string; fall back to default CRS.

-                except Exception:
-                    pass
+                except Exception as exc:
+                    logger.debug("Failed to convert numpy scalar %r using .item(): %s", val, exc)

		if unit_name_field != 'UNITNAME' and unit_name_field in geology_gdf.columns:
		geology_gdf = geology_gdf.rename(columns={unit_name_field: 'UNITNAME'})

-                    pass
+                    # Best-effort conversion to bytes; if this fails, fall back to using the
+                    # original data and let the subsequent fromWkb call handle it.
+                    logger.debug(
+                        "Failed to convert WKB data to bytes in _geometry_from_value",
+                        exc_info=True,
+                    )

	except Exception:
	except Exception:
	# If conversion to a native Python scalar fails, keep the original numpy scalar.

Conversation

rabii-chaarani commented Jan 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

lachlangrose commented Jan 28, 2026

Uh oh!

lachlangrose commented Jan 28, 2026 • edited by rabii-chaarani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lachlangrose commented Jan 28, 2026 •

edited by rabii-chaarani

Loading