Today marks the release of Trajectools 2.3 which brings a new set of algorithms, including trajectory generalizing, cleaning, and smoothing.
To give you a quick impression of what some of these algorithms would be useful for, this post introduces a trajectory preprocessing workflow that is quite general-purpose and can be adapted to many different datasets.
We start out with the Geolife sample dataset which you can find in the Trajectools plugin directory’s sample_data subdirectory. This small dataset includes 5908 points forming 5 trajectories, based on the trajectory_id field:
We first split our trajectories by observation gaps to ensure that there are no large gaps in our trajectories. Let’s make at cut at 15 minutes:
This splits the original 5 trajectories into 11 trajectories:
When we zoom, for example, to the two trajectories in the north western corner, we can see that the trajectories are pretty noisy and there’s even a spike / outlier at the western end:
If we label the points with the corresponding speeds, we can see how unrealistic they are: over 300 km/h!
Let’s remove outliers over 50 km/h:
Better but not perfect:
Let’s smooth the trajectories to get rid of more of the jittering.
(You’ll need to pip/mamba install the optional stonesoup library to get access to this algorithm.)
Depending on the noise values we chose, we get more or less smoothing:
Let’s zoom out to see the whole trajectory again:
Feel free to pan around and check how our preprocessing affected the other trajectories, for example:
A nifty processing toolbox has been added into QField, with countless algorithms available to edit features and geometries while in the field
Geofencing functionality has landed in QField, allowing users to use vector layers to define geofenced areas to warn users or restrict areas where digitizing can occur
Project variables are now shown and customizable within the variable editor
The ratio and resolution settings for photos captured by the QField camera
Date and positioning details can be imprinted onto photos captured by the QField camera
✨ Improvements
A visual revamp of the variables panel provides more space for variable content and easier interaction
QField’s growing documentation is now searchable directly within QField through the search bar, simply type ? followed by keyword(s)
Users can now flag folders as ‘favorites’ when browsing local datasets and projects
The cloud projects list now comes with a nifty search bar
Significant improvements in attachment upload to the QFieldCloud servers (if this was a problem in the past, try again!)
Improvements to the QField camera geotagging functionality
On mobile devices, the home screen has a shutdown button to insure authentication details are cleared
Projects can now have a default active layer set for each map theme
“Help us create the world’s most advanced open database on litter, brands & plastic pollution.”
This sentence welcomes us to the website of the OpenLitterMap project, which aims at providing tools for any and all citizens to capture data on litter worldwide. This data can now be directly accessed in QGIS for visualisation and analysis purposes, thanks to the efforts of the plugin’s author NaturalGIS. Well done to everyone involved, and we wish all the luck to this great project.
Overview
Here follows the quick overview in reverse chronological order. If any of the names or short descriptions catches your attention, you can find the direct link to the plugin page in the table below:
This plugin allows users to directly access SSEC RealEarth web services public catalog of near real-time satellite imagery and related ancillary data through the OGC standard web services of WMTS and WFS. A login to RealEarth is recommended to extend data volume quota before watermarking occurs, but is not required.
Earlier this year, I shared my experience using ChatGPT’s Data Analyst web interface for analyzing spatiotemporal data in the post “ChatGPT Data Analyst vs. Movement Data”. The Data Analyst web interface, while user-friendly, is not equipped to handle all types of spatial data tasks, particularly those involving more complex or large-scale datasets. Additionally, because the code is executed on a remote server, we’re limited to the libraries and tools available in that environment. I’ve often encountered situations where the Data Analyst simply doesn’t have access to the necessary libraries in its Python environment, which can be frustrating if you need specific GIS functionality.
Today, we’ll therefore start to explore alternatives to ChatGPT’s Data Analyst Web Interface, specifically, the OpenAI Assistant API. Later, I plan to dive deeper into even more flexible approaches, like Langchain’s Pandas DataFrame Agents. We’ll explore these options using spatial analysis workflow, such as:
Loading a zipped shapefile and investigate its content
Finding the three largest cities in the dataset
Selecting all cities in a region, e.g. in Scandinavia from the dataset
Creating static and interactive maps
To try the code below, you’ll need an OpenAI account with a few dollars on it. While gpt-3.5-turbo is quite cheap, using gpt-4o with the Assistant API can get costly fast.
OpenAI Assistant API
The OpenAI Assistant API allows us to create a custom data analysis environment where we can interact with our spatial datasets programmatically. To write the following code, I used the assistant quickstart and related docs (yes, shockingly, ChatGPT wasn’t very helpful for writing this code).
Like with Data Analyst, we need to upload the zipped shapefile to the server to make it available to the assistant. Then we can proceed to ask it questions and task it to perform analytics and create maps.
assistant = client.beta.assistants.create(
name="GIS Analyst",
instructions="You are a personal GIS data analyst. Write and rund code to answer geospatial analysis questions",
tools=[{"type": "code_interpreter"}],
model="gpt-3.5-turbo", # or "gpt-4o"
tool_resources={
"code_interpreter": {
"file_ids": [file.id]
}
}
)
Then we can start asking questions and giving our assistant tasks:
thread = client.beta.threads.create()
message = client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content="Please load the zipped shapefile and describe the content"
)
assistant > The shapefile has been successfully loaded as a GeoDataFrame with the following details:
- The GeoDataFrame has 243 entries.
- It contains a total of 38 columns with various data types including integer, float, and object types.
- One of the columns is a geometry column representing the spatial information.
If you have any specific analysis or visualizations you would like to perform on this data, feel free to let me know!
The docs have a nice example for how to stream the answers back, e.g. into a Jupyter notebook. I’ve used it to create a small convenience function:
def ask_another_question(question):
message = client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content=question
)
with client.beta.threads.runs.stream(
thread_id=thread.id,
assistant_id=assistant.id,
event_handler=EventHandler(),
) as stream:
stream.until_done()
ask_another_question("Please list the three largest cities")
assistant > code_interpreter
# Sort the GeoDataFrame by the population in descending order
largest_cities = gdf.sort_values(by='pop_max', ascending=False).head(3)[['name', 'pop_max']]
largest_cities
assistant > The three largest cities in the dataset, along with their maximum population numbers, are as follows:
1. Tokyo: 35,676,000
2. New York: 19,040,000
3. Mexico City: 19,028,000
These cities have the highest recorded maximum population numbers in the dataset. Let me know if you need any further information or analysis!
ask_another_question("extract all scandinavian cities ")
assistant > To extract all the Scandinavian cities from the dataset, we can filter the GeoDataFrame based on the countries typically considered part of Scandinavia. These countries include Norway, Sweden, Denmark, Finland, and sometimes Iceland. Let's filter the GeoDataFrame to extract the Scandinavian cities.
assistant > code_interpreter
# List of countries typically considered part of Scandinavia
scandinavian_countries = ['Norway', 'Sweden', 'Denmark', 'Finland', 'Iceland']
# Filter the GeoDataFrame for Scandinavian cities
scandinavian_cities = gdf[gdf['adm0name'].isin(scandinavian_countries)][['name', 'adm0name']]
scandinavian_cities.head() # Displaying the first few Scandinavian cities
assistant > Here are some of the Scandinavian cities extracted from the dataset:
1. Reykjavík, Iceland
2. Oslo, Norway
3. Helsinki, Finland
4. København (Copenhagen), Denmark
5. Stockholm, Sweden
These are just a few examples of Scandinavian cities present in the dataset. If you would like the complete list or perform any specific analysis on these cities, feel free to let me know!
ask_another_question(
"please create a world map with all cities (use gray circles) "
"with scandinavian cities highlighted as red circles. "
)
assistant > To create a world map with all cities represented as gray circles and Scandinavian cities highlighted as red circles, we will use the GeoPandas library for mapping. Let's go ahead and create the map.
assistant > code_interpreter
import matplotlib.pyplot as plt
# Plot all cities with gray circles and Scandinavian cities with red circles
fig, ax = plt.subplots(figsize=(12, 8))
# Plot all cities in gray circles
gdf.plot(ax=ax, color='lightgrey', markersize=5)
# Plot Scandinavian cities in red circles
scandinavian_cities.plot(ax=ax, color='red', markersize=50, edgecolor='black', label='Scandinavian Cities')
# Set the title
ax.set_title('World Map with Scandinavian Cities Highlighted', fontsize=16)
# Add legend
ax.legend()
# Display the map
plt.show()
assistant > It seems that there was an error while plotting the map because the GeoDataFrame `scandinavian_cities` does not have the necessary numeric data to plot the map directly.
...
plt.show()
output >
assistant > Here is the world map with all cities represented as gray circles and Scandinavian cities highlighted as red circles. The map provides a visual representation of the locations of the Scandinavian cities in relation to the rest of the cities around the world. If you need any further assistance or modifications, feel free to let me know!
To load and show the image, we can use:
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
def show_image():
messages = client.beta.threads.messages.list(thread_id=thread.id)
for m in messages.data:
if m.role == "user":
continue
if m.content[0].type == 'image_file':
m.content[0].image_file.file_id
image_data = client.files.content(messages.data[0].content[0].image_file.file_id)
image_data_bytes = image_data.read()
with open("./out/my-image.png", "wb") as file:
file.write(image_data_bytes)
image = mpimg.imread("./out/my-image.png")
plt.imshow(image)
plt.box(False)
plt.xticks([])
plt.yticks([])
plt.show()
break
Asking for an interactive map in an html file works in a similar fashion.
You can see the whole analysis workflow it in action here:
This way, we can use ChatGPT to perform data analysis from the comfort of our Jupyter notebooks. However, it’s important to note that, like the Data Analyst, the code we execute with the Assistant API runs on a remote server. So, again, we are restricted to the libraries available in that server environment. This is an issue we will address next time, when we look into Langchain.
Conclusion
ChatGPT’s Data Analyst Web Interface and the OpenAI Assistant API both come with their own advantages and disadvantages.
The results can be quite random. In the Scandinavia example, every run can produce slightly different results. Sometimes the results just use different assumptions such as, e.g. Finland and Iceland being part of Scandinavia or not, other times, they can be outright wrong.
As always, I’m interested to hear your experiences and thoughts. Have you been testing the LLM plugins for QGIS when they originally came out?
With the QGIS Grant Programme 2023, we were able to support six proposals (four in the first round and two in the second round) that are aimed to improve the QGIS project, including software, infrastructure, and documentation. The following reports summarize the work performed in the first four proposals:
Existing Processing algorithms Voronoi Polygons and Delaunay Triangulation have been ported to C++ and now use GEOS instead of the unmaintained Python module.
Two algorithms for generating XYZ tiles (directory and MBTiles variants) have been ported to C++ using a safer and cleaner multi-threading approach.
The Align Rasters tool, which was not exposed to Processing, has been removed and a new Processing algorithm with the same functionality has been added.
The existing Raster Calculator algorithm has been ported to C++. The algorithm now has two variants: a toolbox version that works the same way as before, and a modeler version that uses the same approach to input naming as the GDAL raster calculator.
As of QGIS 3.34, QGIS can now create and handle vertical and compound CRSes.
In QGIS 3.34 coordinate transforms were reworked so that they function correctly with vertical transformation, if both the source and destination CRS have vertical components.
In QGIS 3.36 the coordinate reference selection widgets were updated to offer choices of 2d only, compound, or vertical only CRSes.
In version 3.38, we introduced a new setting for QGIS projects, for their vertical reference system. Users can control this through project properties, and it’s accessible via PyQGIS and via associated expression variables (eg @project_vertical_crs) for use in print layouts.
Similarly, in 3.38 we introduced the API support for map layers to have a vertical CRS. (This was not exposed to users in 3.38 though)
In QGIS 3.40 so far we have exposed the vertical CRS setting for vector layers to users (via the layer properties dialog), allowing users to specify the associated vertical CRS for these layers. The vertical CRS is respected in elevation profile plots, in Identify tool results, and in 3D Map views (assuming the 3D map is created with an associated vertical CRS).
There is an open pull-request for 3.40 to expose the vertical CRS for point cloud layers in a similar way, with the vertical CRS being respected in elevation profiles, identify tool results, and 3D map views
We have open pull requests for 3.40 to show layer vertical CRS information in the layer properties “information” pages, and add expression variables at the layer scope (eg @layer_vertical_crs).
Improve test result handling on QGIS CI (#268)– Report Any tests which fail a rendering comparison will write a descriptive comment to the PR. The comment details which render tests failed, where they are in the code, and includes some helpful pointers to downloading the full test report and the QGIS developer documentation. We also implemented lots of improvements in running the tests locally and how the render test reports are generated and presented to developers.
Thank you to everyone who participated and made this round of grants a great success and thank you to all our sustaining members and donors who make this initiative possible!
For the last month of July, one short of 30 new plugins were published in the QGIS plugin repository.
Here follows the quick overview in reverse chronological order. If any of the names or short descriptions catches your attention, you can find the direct link to the plugin page in the table below:
DeraPro for projects and studies allows the download of the reference spatial data of Andalusia and make a cut of the information for a specific study area.
The CBERS Explorer is a specialized plugin developed to facilitate the search and retrieval of images from the China-Brazil Earth Resources Satellite (CBERS) program, specifically focusing on the CBERS-4A satellite with Wide Field Imager (WPM) data.
A comprehensive toolset for coordinate calculations, conversions, spatial operations, watershed analysis, land use analysis, and flood simulation in QGIS.
Tworzy słownik w formie pliku txt rozdzielanego tabulacjami dla wybranych warstw, program pozwala na eksport pełnych nazw pól danej warstwy oraz wersje jako powstanie po zmienie z gpkg do shp.
The use of WMS/WMTS layers in a QGIS Cloud map project can significantly degrade the performance of the map display. I have already discussed how to counter this problem in an earlier post. One of the solutions is to load external WMS as background layers. The problem with this approach, however, is that only one WMS background layer can be loaded at a time. If further WMS layers are to be loaded into the map at the same time, this approach cannot be used.
We recently launched our new website at QGIS.org. It is a ground-up overhaul and provides a fresh take on the first contact point for existing or potential users wishing to engage with the QGIS project and discover its value proposition.
A new strategy for QGIS.org websites
In this blog post, we would like to provide an overview of the goals that we had for building the new QGIS.org website and the bigger picture of how this website update fits into the broader strategy for our website plans for QGIS.
About two years ago, we started experimenting with building a new QGIS.org website based on Hugo. Hugo, as a technology choice, was less important than was our intent to develop a more modern site that addressed our strategic goals.
After some ‘in-house’ (i.e. volunteer-based) work to develop an initial version of the site, we received the go-ahead to use QGIS funds for this and put out a call in October 2023 for a company to support our work. This was ultimately won by Kontur.io, who, together with our volunteers, brought the work into high gear.
Initial analysis of the questions and actions to be quickly answered by qgis.org
Goal 1: Speak to a new audience
Our primary goal was to speak to a new audience. We are confident that QGIS can compete with all of the commercial vendors providing GIS software. We didn’t convey that well on our old website. We feel that QGIS was too apologetic in how it presented itself. We wanted a website which inspires confidence while addressing the needs of a corporate or organisational decision-maker who is looking at the QGIS project during their GIS software selection process.
The old website was very focused on the developer and contributor community. Obviously, those aspects are important since, without our fantastic community, the QGIS project would not exist. The messaging around open source is also important. Yet these ideas are secondary to the idea that QGIS is one of the best (if not the best) desktop GIS applications out there on the market – open-source or otherwise. We need to present it in this professional perspective.
So, the first goal was to change the messaging to focus on QGIS’s value proposition and take a very professional approach to presenting ourselves on the website.
User group and requirements analysis for the potential qgis.org visitors
Goal 2: Harmonisation
The second goal was to start the process of harmonising all of our website properties. QGIS.org, over the years, has built many different web properties. For example, there’s the plugins website, the feed, the changelog, the sustaining members website, the lessons website and the certification website, the new resources hub website, the API documentation, the user documentation, the user manual, the training manual, various other documentation efforts, and more. Some of those are combined in one application, There are also some less well-known resources, like our analytics.qgis.org and another one for plugin analytics. In short, we’ve a lot of resources!
With so many different web properties, they’ve devolved over time: each has its own look and feel, navigation approach and how you interact with it. Some of them were translated, and some of them were not. We want to harmonise all of these sites so that the user does not notice any change in user experience when they move from one QGIS-related site to another.
Goal 3: Harmonising deployment
In the underlying process of these changes, we’re also redeploying all of the websites on new servers, which are more up-to-date and use better security and maintenance practices. Plenty of work is happening in the background to ensure that all of the servers are in a better-maintained state, document how they’re maintained, and so on.
Goal 4: A hub and spokes
The objective of the new site design is to allow quick movement between the QGIS auxiliary sites. The QGIS.org site will form a hub that effortlessly takes visitors to whichever QGIS-related site they need to complete the task they are busy with. If you’re moving between these sites, the experience should be seamless. You should not really even be aware that you’re moving between different websites. Other than looking at the URL bar, the user presentation and experience should be harmonious between all of them.
One way we are planning to achieve this is to have a universal menu bar and footer. You will see that in the new website’s design, there is a menu bar across the top. This menu bar has two levels: the top menu and the second level, where the search bar is.
The universal menu bar
In this second row, auxiliary sites will have their own sub-menu whilst keeping the shared top-level menu. So if you, for example, are moving around in plugins and want to review the plugin list or submit a new plugin, all of that navigation will be on the second line where the search bar is currently. Regardless of which subdomain you are on, the top-level menu bar will be the same, allowing you to easily navigate back to the hub or to another subdomain.
The footer will be unified and shared between all sites, and the cascading style sheets and styling will be unified across all of the QGIS websites.
In the next phase, we will work to achieve this coherence across all the websites, though we still have a few more tweaks to make to the qgis.org site first.
Goal 5: DOTDOTW – do one thing, do one thing well
We plan to break some auxiliary websites apart into separate pieces. So, for example, the changelog management, certification management, sustaining members management, and lessons management are all in one Django app. We will split them into small single-purpose applications using some common UX metaphors so that each is a standalone application that makes it easy for a potential contributor to understand everything the application does. This will also simplify management as we can upgrade each auxiliary site on separate development cycles. We will also finally have semantic URLs, e.g. certification.qgis.org, to take you to the different areas of interest on the site.
The plugins.qgis.org is also going to be refactored so that it just has plugins and not the resource sharing we’ve added in the last few years. The resource sharing will go into its own subdomain. Similarly, the Planet website will get split into its own website (the planet is a blog aggregator or RSS aggregator) that will be in its own managed instance. Some other components (like the analytics) are difficult to split out like this because they’re linked to the same database. We will try to make sure that those are more discoverable and theme them as much as possible to match the rest of the website experience.
Goal 7: Encapsulation
Another goal we had for the QGIS.org makeover was to make the site performant and self-contained. By self-contained, we mean that it should not ‘call’ out to CDN, Google or other platforms for resources like fonts, CSS frameworks, javascript libraries, etc. There were two reasons for this:
These platforms often use such resources to track users as they move around the Internet, which we want to avoid as much as possible.
We want to wholly manage our site, be able to fix any issues independently and generally follow a path of self-determination.
Our approach also facilitated the creation of a very performant website, as you can see here. We will try to adhere to these principles for the auxiliary site updates we do in the future, too.
What about translations?
The question has come up: Why did we not want to translate the new QGIS.org when it was translated before?
Firstly, we should make it clear that we do not plan to remove translations from the user documentation, the user manual, and so on, where we think they have the most value.
For the main QGIS.org site, we question whether there is a high value in translating it. Here are some reasons why:
1. Lingua franca: If you are an IT manager in a non-English-speaking country and you want to evaluate some software, you’re going to run into a product page that presents itself in English – it is the norm for IT procurement to work in English for reviewing software products and so on.
2. Automation: Automated translations inside browsers are getting better and better. While these translations are still not completely adequate, we think they will be in one or two years’ time.
3. Translation integrity: Our pursuit of Goal 1 means that we would no longer find it acceptable to have partial website translations. We also need to ensure that the wording and phrasing are consistent with the English messaging. We also have concerns about the QA process regarding trust and review – we want to ensure that any translation truly reflects the meaning and intent of the original content and has not been adjusted during the translation process.
4. Cohesion: Our most important point is raised if we go back to this idea of cohesion between the different websites like QGIS.org, plugins.qgis.org and so on. As well as having the same styling, we also don’t want to switch between languages as you hop between the sites. We aim to present them all as one site. If we translate QGIS.org and then take you to our auxiliary sites, e.g., plugins.qgis.org, the feed, or certification pages, which are in English only, the experience is jarring.
So we must either translate everything into all of the same languages, or work in English. Translating everything is a mammoth task for the translators and for us to retrospectively add translation support to each platform. Thus, we prefer the approach of harmonising everything to one language and then focusing our translation efforts on three areas:
The application itself,
the user manual and
the training manuals.
We can leave the rest of the experience in English and instead focus on harmonising, for now, both in terms of look and feel and the technology used.
When we consider everything as one big website and what the bigger plan is, it is hopefully clearer why we didn’t think translating the landing page and QGIS.org was the best approach.
Further funded work
We hope to use more QGIS funding to support this work in the future. We’re also hoping to work again with Kontur to start moving all these auxiliary sites into their own projects, applying our style guidelines to each. Independently of that, Tim (volunteer), Lova (QGIS funded), and others are already getting started with this process.
Helping out
Do you have strong opinions about the website? Contact Tim on the PSC mailing list if you would like to get involved as a volunteer. We would love to hear from designers, word smiths, marketers, information architects, SEO specialists, web developers and those who think they can help us achieve our goals.
Conclusion
We hope our goals and process make sense for everybody and that we were able to lay out a clear, logical argument about why we don’t want to translate the new website quite yet. We want to focus on these overarching goals and then return to them later if they are still a priority for people. Everything we have built is Open Source and available at this repo, where you can also find an issue tracker to report issues and share ideas relating to the new website.
Here follows the quick overview in reverse chronological order. If any of the names or short descriptions catches your attention, you can find the direct link to the plugin page in the table below:
