---
title: "Editing GTFS feeds"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Editing GTFS feeds}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
# the queries in this vignette use {dplyr}, which gtfstools only suggests
has_dplyr <- requireNamespace("dplyr", quietly = TRUE)

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = has_dplyr
)

Sys.setenv(OMP_THREAD_LIMIT = 2)
data.table::setDTthreads(2)

library(gtfstools)
```

GTFS feeds describe the transit services of a city or region as they are. Quite often, however, one wants to analyze how things would be if they were different: what if a bus route ran more frequently, or if a rail line were faster? Answering such questions requires editing the feed, changing its service levels, travel times or the way its schedules are described, and then feeding the result to routing and accessibility tools.

**gtfstools** includes several functions to edit GTFS feeds. The editing functions currently available are:

- `set_route_frequency()`: sets the headway of routes within a time of day;
- `set_trip_speed()`: sets the average speed of trips, or of a segment of them;
- `stop_times_to_frequencies()`: approximates scheduled trips by frequency-based ones;
- `interpolate_stop_times()`: fills blank stop times, assuming a constant speed between stops with known times;
- `set_route_speed()`: sets the average speed of the trips of routes, or of a segment of them;
- `set_max_speed()`: caps the speed of trips, slowing down only the parts that are faster than a maximum speed;
- `set_dwell_time()`: sets how long vehicles stay at stops;
- `frequencies_to_stop_times()`: converts frequency-based trips into scheduled ones;
- `merge_gtfs()`: combines several feeds into a single one;
- `remove_duplicates()`: removes duplicated entries from the feed's tables;
- `remove_unused_ids()`: removes the ids that the feed doesn't use, with their entries in every table;
- `fix_gtfs()`: cleans a feed in one call, removing duplicated entries and unused ids, optionally capping the speed of trips and filling blank stop times.

This vignette will introduce you to the first four functions and will cover their usage in detail. By default, they return an edited copy of the feed and leave the given GTFS object unchanged.

We will start by loading the packages required by this demonstration into the current **R** session:

```{r, message = FALSE}
library(gtfstools)
library(dplyr)
```

# Setting route frequencies

The number of departures of a route is one of the main levers of a transit network, and one that is frequently changed in scenario analyses. `get_route_frequency()` tells us how many departures each route has within a time of day, and its mean headway. Let's see how frequently the route `T2` of Porto Alegre's feed runs during the morning peak:

```{r}
path <- system.file("extdata/poa_gtfs.zip", package = "gtfstools")
gtfs <- read_gtfs(path)

get_route_frequency(
  gtfs,
  route_id = "T2",
  from = "07:00:00",
  to = "09:00:00"
)
```

The route runs on three different services (`T2@1` on weekdays, for example), each with a different headway. `set_route_frequency()` sets a single headway, in minutes, to all of them. Let's say we want the route to run every 10 minutes from 7 to 9 am:

```{r}
new_gtfs <- set_route_frequency(
  gtfs,
  route_id = "T2",
  headway = 10,
  from = "07:00:00",
  to = "09:00:00"
)

get_route_frequency(
  new_gtfs,
  route_id = "T2",
  from = "07:00:00",
  to = "09:00:00"
)

new_gtfs$frequencies
```

Under the hood, the function replaces the trips that depart within the time of day by a single template trip per service (and per direction, when `direction_id` is present in `trips`), taken from its most common sequence of stops. The departures of the template are described in the `frequencies` table. Since this feed had no `frequencies` table, one has been created. The template keeps its travel times between stops, which apply to all of the new departures. Departures outside the time of day are kept as before. Please note that departures after midnight are listed with times greater than `"24:00:00"`, so a time of day such as `"24:00:00"`-`"26:00:00"` should be used for them.

The trips removed by the function may leave unused entries in other tables, such as `shapes` and `calendar`, which can be removed with `remove_unused_ids()`. If your routing software doesn't support frequency-based trips, `frequencies_to_stop_times()` converts them back into scheduled trips.

# Setting trip speeds

`set_trip_speed()` sets the average speed of trips by changing the `arrival_time` and `departure_time` columns in the `stop_times` table. Let's see how that works using the trip `CPTM L07-0` of São Paulo's feed. This trip is described in the `frequencies` table, so its times are just a "template" that describes how long it takes from one stop to another, relative to each of its departures.

```{r}
path <- system.file("extdata/spo_gtfs.zip", package = "gtfstools")
spo_gtfs <- read_gtfs(path)

get_trip_speed(spo_gtfs, trip_id = "CPTM L07-0")

# 'unit' argument defaults to "km/h"
new_speed_gtfs <- set_trip_speed(spo_gtfs, trip_id = "CPTM L07-0", speed = 50)

get_trip_speed(new_speed_gtfs, trip_id = "CPTM L07-0")

new_speed_gtfs$stop_times |>
  filter(trip_id == "CPTM L07-0") |>
  select(stop_id, stop_sequence, arrival_time, departure_time)
```

The duration of the trip is set to its length, measured along its shape, divided by the given speed. `set_trip_speed()` updates the arrival time at the last stop accordingly and sets the arrival and departure times at intermediate stops to `""`. To fill in the expected arrival and departure times of intermediate stops, you use `interpolate_stop_times()`, as we'll see in a [later section](#interpolating-stop-times).

The `set_trip_speed()` function also includes a few other arguments that control which trips and parts of trips are changed:

- `unit` sets the unit in which the speed is given, either `"km/h"` (the default), `"m/s"` or `"mph"`;
- `first_stop` and `last_stop` set the speed of only a segment of the trips, between two stops. The times at the stops after the segment are shifted by the change in its duration;
- `from` and `to` only change the trips that depart from the segment's first stop between `from` "HH:MM:SS" and `to` "HH:MM:SS" (both included);
- `by_reference = TRUE` updates the `stop_times` table by reference, thus changing the given GTFS object.

# Converting stop times to frequencies

Some routing engines and other tools treat frequency-based trips differently from scheduled ones. For example, the `time_window` parameter of [`{r5r}`](https://ipea.github.io/r5r/) draws departure times within the headways of frequency-based trips. `stop_times_to_frequencies()` converts scheduled trips, described only in `stop_times`, into frequency-based trips, described in `frequencies`. Let's convert all the trips of Porto Alegre's feed:

```{r}
converted_gtfs <- stop_times_to_frequencies(gtfs)

nrow(gtfs$trips)
nrow(converted_gtfs$trips)

# frequencies of route T2 on weekdays, from 6 to 10 am
converted_gtfs$frequencies |>
  filter(
    startsWith(trip_id, "T2-1@1#"),
    start_time >= "06:00:00",
    start_time < "10:00:00"
  )

# the number of departures of route T2 is kept
get_route_frequency(
  converted_gtfs,
  route_id = "T2",
  from = "07:00:00",
  to = "09:00:00"
)
```

The trips that share the same route, service, direction, shape and sequence of stops are grouped together and split into one-hour slots of the clock, according to their earliest departure time. In each slot, the trip that departs first becomes a template trip, and a `frequencies` entry is added for it, from the start to the end of the slot, with a headway of `ceiling(3600 / n)` seconds, `n` being the number of trips in the slot (e.g. the 9 trips of route `T2` that depart between 7 and 8 am on weekdays give a headway of 400 seconds). The other trip ids in the same template are removed from the `trips.txt` file, but it keeps the same number of departures.

Please note that this is a lossy approximation of the original schedule: the exact departure times of the removed trips, their travel times and other fields in `trips` (e.g. `trip_headsign`) are lost. Converting the result back with `frequencies_to_stop_times()` yields the same number of trips in each slot (as long as it has at most 60 trips), but their departures are evenly spaced from the start of the slot. To convert only some trips, pass their ids to the `trip_id` argument.

# Interpolating stop times

Sometimes, GTFS feeds don't report the arrival and departure times at every stop: only the times at the first and last stops of each trip are required, and the stops in between may be left blank. The stops with known times are called timepoints. In Porto Alegre's feed, for example, only the first and last stops of each trip have times:

```{r}
trip <- "T2-1@1#520"

gtfs$stop_times |> 
  filter(trip_id == trip) |> 
  select(stop_id, arrival_time, departure_time) |> 
  head()

```

`interpolate_stop_times()` fills these blank times, assuming that the vehicle travels between two consecutive timepoints at a constant speed, with distances measured along the trip's shape:

```{r}
interpolated_gtfs <- interpolate_stop_times(gtfs, trip_id = trip)

interpolated_gtfs$stop_times |> 
  filter(trip_id == trip) |> 
  select(stop_id, arrival_time, departure_time) |> 
  head()

```

Interpolated stops get equal arrival and departure times (i.e. no dwell time), rounded to the nearest second, and `timepoint = 0` when the `stop_times` table includes a `timepoint` column. In feeds without shapes, use `method = "euclidean"` to measure distances as straight lines between stops instead.

If `trip_id` is not given, the function interpolates the data to give explicit stop times at every stop of every trip in the feed. Some stops may be left blank, with a warning: for example, a few trips of Porto Alegre's feed list their times after midnight as `"00:xx:xx"`, instead of `"24:xx:xx"`, so they seem to arrive before they depart and can't be interpolated.

**Note:**

All editing functions return a GTFS object readily available to be manipulated and analyzed using the rest of **gtfstools**' toolkit, or to be written to disk with `write_gtfs()`. For more information on how to use other functions made available by the package, please see the [introductory vignette](gtfstools.html) and the [function reference](https://ipea.github.io/gtfstools/reference/index.html).