Blog

  • Terminology Guide:
  • Tools - now resembling MS Office
    • Watch out for launcher for more tools
  • Use MXDs and import
    • Insert
    • Find MXD
    • No more backwards compatibility issues
    • No pain - just insert!
    • Pro back to MXD?  Not possible (just oneway)
  • Key Concept:  Project
    • Data, 
    • Maps
    • Layouts settings
    • Template-driven
  • Locate Pane
    • Enhanced
    • Customizible 
  • Project Packages
    • Easy sharing
  • Configuration/customization
    • Check out ICAP - for customizing the start-up
    • In the 'Backstage' area (click Project)
    • Very similar to MS Office, but with an ESRI twist
    • Combines a lot of the features from ArcGIS Administrator - all of it is now in the Backstage
  • Map views can be docked in diffferent window!
  • Very focused on Web Services now
    • Connecting to ArcGIS Online/Portal
  • Editing:  Always on
    • No more starting/stopping an edit session
    • Linked views are possible - 2D+3D
  • Enhanced Definition query!
    • Multiple DQs - enable the active one you want
  • ArrTools:
    • Search is embedded
    • History (when a green checkbox is present
    • Interface is integrated into panel/pane

Introduction to Tableau Desktop Pro

Are you new to Tableau? The two-day class will cover the fundamentals which includes: checking for data accuracy, creating visualizations, combining visualizations in an interactive dashboard, and sharing dashboards with Tableau Server.

Intermediate Training for Tableau Desktop Pro

Are you using Tableau regularly and ready to learn more? OCTO’s two-day intermediate Tableau course is designed for analysts and District of Columbia agency staff who have a solid working experience with Tableau and want to take it to the next level. You should have a deep understanding of all the fundamental concepts of building worksheets and dashboards, but may have trouble when working with complex issues. The training is for DC government full-time employees only.

Training location

The training will be hosted at OCTO’s 200 I Street, SE office in room 5009. Please arrive by 8:45 a.m. to go through the building’s security and be escorted to the 5th floor.

Fee

No charge; included in OCTO’s BI program.

Agendas 
Must be connected to DC.gov to view

 

Meals will not be provided; there are a variety of lunch options within walking distance.

Please contact Tim Harwood with any questions at tim.harwood@dc.gov

ESRI UC 2018 RoundUp

Python API

Reference to: https://developers.arcgis.com/python/guide/

Type 'import this' in python!

Jupyter Notebooks

  • Execute python from within a Web IDE.
  • Presentation Ryse for Jupyter notebooks for python presentations - demo code/scripts.
  • Allows users to visualize data/results
  • Use pandas for display table dataframes for visualization
  • Visualization as script goes... super useful
  • Use period + tab for autocomplete
  • Shift+tab to find out info about functions (parameters, etc)
  • Question mark after the cell gives you the docstrings of the object.
  • To add layers, 

Authentication for API for Python

  • Anonymous - you can search for data, add layers, etc.
  • Built-in users:  create dict of multiple orgs if needed
  • For SSO - need to create an appID
    • Go to Content in AGO
    • Create add item - application.
    • Give it a name
    • In settings, register app.
    • Use App ID to authenticate
    • Check user docs to see how to authenticate
    • Dashboards using Python API

Analysis

  • Awesome feature detection

Hub for ArcGIS 


  • Hub helps you organize and engage communities of interest around focused initiatives
  • Engagement Maturity Path.
    • Open Data Phase1:
      • One-way policy of communication/ad hoc datasharing
      • Then, Community self-service is/point solutions
      • Then, Addhoc community involvement/Formalized Open Data
    • Initiatves
      • THen, Initiative-focused information/organizing apps, solutions and analysis around an initiatives
      • Then, Sustained 2-way engagement/frequent data contribuion from community
    • Fully engaged:
      • Finally (the major goal) Community members regular partners/Community creates analyses and apps/sollutions
  • Initiative Pages have common components
    • Panner
    • Narrative
    • Performance
    • Apps
    • Data Catalog
    • Paps
    • Ways to Engage
    • Participating Agencies
  • Initiative templates are available to start
  • What is it?
    • Bundle of ArcGIS capaabililities 
      • Site
      • Survey
      • Event
      • Storymaps
      • Data
      • Dashboards
  • VZ will be a template!
  • Community engagement seems to be the big differentiator with regard to Hub
  • Users can create accounts in Hub with their Google/Fb account
    • They can contribute to the initiative.  (this is one of the big concerns with OCTO - fear of managing a site that public has accounts on.  Still need some clarity about this)
    • You can 'promote' the citizens if they are providing good feedback/help.
    • Accounts are connected to, but are completely separate from your ArcGIS Org.  This ensures that your Org is isolated and secured.
  • Redesign of Events system is coming soon. (2-3 months)
  • Commenting and discussion improvements also coming soon.

Note

This blog post was drafted in preparation for an upcoming FHWA GIS in Transportation Newsletter. This article can be found at:

https://www.gis.fhwa.dot.gov/documents/Newsletter_July2018.asp

Introduction

At the 2017 FHWA Highway Information Seminar (HIS), representatives from several states gathered to discuss a great variety of topics:  roadway GIS inventory, spatial analysis, traffic data modeling, Big Data, pavement data and a multitude of other topics.  One of these discussions involved the future of data collection and how states might implement new approaches to make the process more efficient.  The heart of the conversation was that programs should strive to 'make things once and use them a bunch'; GIS data should fulfill a multiplicity of uses while reducing 'one-off', static data products.1

The Model Inventory of Roadway Elements (MIRE) Report is a 'comprehensive listing of data elements needed for safety analysis'.2  Looking beyond its rather staggering size (there are over 200 individual data items), the District Department of Transportation (DDOT) sees MIRE as an opportunity to take a look at our systems of record (namely, the linear referencing system, or LRS) and reflect how we can improve them to 'make things once, then use them a bunch'.

This article suggests two primary approaches to aid in the automated extraction of a large number of MIRE data elements (81, estimated), however the benefits go beyond MIRE. DDOT contends that the Linear Referencing System (LRS) centerline should be regarded as more than simply 'where the centerlines come from'.  Taking a new look at what's possible with LRS can yield additional data in support of MIRE and potentially much more.  

Background:  FHWA's MIRE Report

The 'MIRE Report' ('MIRE 1.0') lists individual safety data elements and describes the data collection requirements for those elements.  The report states unequivocally:  simply having high quality crash data is not enough.  Detailed GIS roadway and traffic data are essential elements, enabling a more comprehensive safety data analysis and providing more context and understanding to the entire safety picture.  According to the MIRE Report, "(t)he Interactive Highway Safety Design Model (IHSDM), SafetyAnalyst, the Highway Safety Manual (HSM), as well as the National Cooperative Highway Research Program (NCHRP) Series 500 Data and Analysis Guide, all require crash, roadway, and traffic data to achieve the most accurate results".2  

MIRE 2.0

In early February 2018, the MIRE 2.0 Report was released3.  Below are a few of the major changes:

  • 3 additional elements (2 items were split; 1 new item)
  • Each element has a very useful 'crosswalk' table.  This table notes where what other system or program this data item can be imported from.
  • One rather significant retraction:  FHWA no longer indicates which items are optional or required by the above-referenced safety programs/systems. 

Overall, the updates are a welcome change:  bringing useful references to help point DOTs in the right direction for building their inventory of safety data for analysis.

MIRE FDE

In 2011, FHWA classified 37 data elements within MIRE as critical for safety data analysis:  Fundamental Data Elements or FDE.  The FDE were classfied as 'essential' items needed to meet the basic requirements of most any safety analysis system, including the safety analysis tools mentioned above.4  Additionally, the MIRE FDE are a requirement established by the MAP-21 legislation and reaffirmed by the FAST Act legislation.  With some minor adjustments in MIRE 2.0, the total number of FDE now rests at 38 data elements (Federal Aid/Route Type was split into two elements).  Thankfully, the true count of the FDE is actually less than that:

  • Duplicated MIRE Elements:  Of these 38 FDE, four elements are duplicated, appearing twice (but in different sections). This brings the total unique elements down to 34.  The duplicates are:
    • Functional Class
    • AADT
    • AADT Year 
    • Governmental Ownership.  
  • Already Covered by HPMS.  We found that of those remaining 34 MIRE elements, nearly half (15) are satisfied by a statewide HPMS full-extent submission. 

The remaining 19 non-HPMS FDE shown in Table 1 below should then become the central focus of the state's responsibility as it relates to MIRE. 


MIRE NumberDescription
18Direction of Inventory
21Federal Aid
24Surface Type
55Median Type
110Unique Junction Identifier
112Location Identifier for Road 1 Crossing Point
113Location Identifier for Road 2 Crossing Point
116Intersection/Junction Geometry
121Intersection/Junction Traffic Control
129Unique Approach Identifier
168Unique Interchange Identifier
172Interchange Type
177

Ramp Length

181Ramp AADT
182Year of Ramp AADT
185Roadway Type at Beginning Ramp Terminal
187Location Identifier for Roadway at Beginning Ramp Terminal
189Roadway Type at Ending Ramp Terminal
191Location Identifier for Roadway at Ending Ramp Terminal

Table 1:  MIRE FDE Which are Not Satisfied by HPMS Data

Making MIRE More Sustainable

The goal of this project is to maximize the automated extraction of the MIRE elements to the greatest extent possible.  From DDOT's perspective, there are two main factors which inhibit the automation:

  1. Limited Guidance for 'Linking' Related Data.  Throughout the MIRE Report (both 1.0 and 2.0) a multitude of MIRE elements mention that the item "...(m)ust be consistent with other MIRE files for linkage."  Although it is only a suggestion, we view this as an absolutely critical requirement.  In his presentation from the 2017 GIS for Transportation Summit, Jesse Day stated "what MIRE needs most is a data model"5.  Many of these data are heavily interrelated.  Without a data model to relate and link the vast array of data items, MIRE data can be difficult to manage and collect.
  2. Existing Roadway Data Sources Require Continuous Collection and Recollection In FHWA's MIRE MIS Lead Agency Data Collection Report6, the efforts of two states highlight many of the challenges currently facing most states.  The two states (New Hampshire and Washington) provided very good approaches to the problem by deriving data from GIS (or other systems) and also performing field and in-office collection to capture remaining MIRE data elements.  But one major issue remains:  if subsequent changes are made to the roadway, many of the MIRE elements would need to be recollected.  Collecting data in this way is difficult to sustain because the MIRE elements are collected and maintained individually.  To the greatest extent possible we should derive and extract from GIS/LRS.  If the LRS does not have the data which allow for the automated extraction of MIRE elements, we should seek to add high-value datasets to our LRS which do.

To address these issues, DDOT has chosen two approaches which when used in combination, have helped us automate a significant quantity of LRS-referenced MIRE data items:  

  • Intersection Model:  Create a simple intersection model which maintains key topologic relationships between the intersection and roadways, derived from the LRS.  
  • Roadway Cross-section Data: Create a sectional LRS inventory of the road and include relevant types of information which are commonly needed by MIRE, such as lanes, buffers and medians, widths and directionality.  

MIRE Intersection-Leg Model

The MIRE Report gives special attention to at-grade intersection/approaches, which comprise over one quarter of the total MIRE listing (58).  Most of these elements are relational in nature.  For example, in MIRE 2.0, element #147 is Crosswalk Presence/Type.  Simply collecting the crosswalk location is not sufficient because a link/reference ID must be stored to persist the relationship between the crosswalk and the Intersection Approach ID (MIRE #128); similarly, a link must also be stored between the intersection approach and the main intersection point which is MIRE element #110, or Unique Junction Identifier

To account for this complexity, it is preferable to maintain a data model that can accomodate the complexity.  DDOT created a very basic 'Intersection-Leg' data model which addresses our need in this regard:  a single point represents an at-grade intersection; LRS event segments are created to relate the intersection to the intersecting routes.  (NOTE:  For this project, we chose to focus on modeling for the at-grade intersection types, but have plans to address interchanges in a secondary project).

Below are the primary components for this basic intersection model:

  1. Intersection Points
  2. Roadway Segments
  3. Intersection Approaches

Intersection Points

ESRI Roads and Highways (DDOT's enterprise LRS platform) manages intersections as a collection of points - two or more points for each intersecting road at that intersection.  For MIRE, a single, unique entity/point should represent the intersection.  Similar to other DOTs, we have authored a script process to reduce this collection of points to a single point feature to serve as the representative point for the intersection7.  This script reduces the stack of intersection points (#1 as shown in Figure 1) into a single representative point (#2).

Figure 1: Reducing Roads and Highways Intersections to Single Intersections


Roadway Segments

A LRS is typically designed with centerlines as single, unbroken route polylines. For DDOT, a Roadway Segment is a segmented route polyline which falls between two intersection points.  This is somewhat similar to the GIS process of 'planarizing' line features, but instead of splitting polylines at every intersection, DDOT requires that driveways and other private road polyline types are ignored.  Because DDOT needs to be selective about what necessitates a break in the geometry, we've created a script that does just that7. Figure 2 below illustrates the segmentation results.  Note that the private driveway (the green dashed line), has been excluded; because of this, segment #1 remains unbroken.

Figure 2: Illustration of roadway segmentation


Intersection Leg Segments
Leveraging both the Intersection Points and Roadway Segments, it is then quite straightforward to create 'leg' segments for a given intersection.  In final script for the intersection model, we create a series of linear events (up to 10 meters in length), offset from the related intersection point for every connected Roadway Segment7.  Figure 3 below illustrates the results at an example intersection.  Once the events are created, we persist the azimuthal direction and all relevant topological relationships into the attribute table.  Another purpose for the Approach Segments in this context is to provide a discrete linear selector for roadway cross-section event data (discussed below in Extraction).

Figure 3: Generating Intersection Legs at Intersections

Other Intersection Modeling Tools

While the above may work for DDOT, there are a few commercial firms that offer much more advanced LRS-based intersection modeling tools for LRS intersection creation and MIRE data management.  DDOT plans to use Intersection Manager8 from Transcend Spatial Solutions to help us manage and extract MIRE data for interchanges.

Cross-sections, Directionality and Relationships

Having an intersection model is a great start to organizing and structuring the LRS for MIRE, but a model alone will not make much of a dent in the MIRE data elements.  You must have a rich data source that lends itself to many uses.  At the start of the article, we stated that an overarching goal should be to 'make it once, use it a bunch.'  Looking at the MIRE elements with this perspective in mind, we attempted to identify elements which are amenable to automation.  We noticed that many elements were simple descriptions, having one or more of the following three key characteristics:

  1. Cross-sectional roadway data
    1. Lane type counts
    2. Width or length 
    3. The presence/absence of a lane type or feature.
  2. Directionality.  Traffic flow/direction in relation to one or more of the above.
  3. Relationships.  A topological relationship between an intersection and/or a connected roadway

Cross-section Events

Looking at the above key items, we determined that capturing a linearly referenced 'high-definition' cross-section of the roadway would capture a significant percentage of MIRE and potentially benefit other program needs well beyond MIRE.  Since LRS is our system-of-record and maintaining a connection to that system is highly valued, DDOT determined that every element of the cross-section should be stored as a linear event within the LRS.  Additionally, the events are indexed from the left to the right, according to the ascending geometry direction of a given route.  A cross-section event can be any number of features that occupy a meaningful portion of the roadway, such as:  through lanes, turn lanes, medians, parking lanes, painted buffers, bike lanes, or a painted centerline.  This is very similar to the graphical cross-sections often used by transportation planners and designers (see Figure 4).  For a comprehensive listing of the cross-section elements DDOT collected, please visit our detailed data dictionary on the DDOT GIS wiki site9.  

Primary attributes for the DDOT LRS cross-section include:

  • Index Position (left to right as geometry ascends)
  • Cross-Section Types
    • Lanes
    • Medians
    • Barriers and Buffers
  • Direction of Section (Lanes only)
  • Width (ft)

Roadway Context

A key point to highlight is that this method captures all section features as they appear in the roadway from left to right.  Capturing the data in this way preserves the 'context' of the roadway as a core component of the data.    For example, it's easy to determine from the data whether or not a bike lane is directly adjoining a parking lane.  Referring to Figure 4 below, one can tell the bike lane in the graphic is index 5; since the parking lane is index 6, they are adjoining.  Table 2 below shows how Figure 4 looks like in the LRS cross-section event table. 

Figure 4: Cross-section of indexed LRS data using Streetmix

                      


LRS Cross-Section Index

1234567
TypeParkingThroughDouble-Yellow CenterlineLeft-Turn OnlyThroughBikeParking
Width (ft)810--101068
DirectionInboundInbound--OutboundOutboundOutboundOutbound

Table 2: LRS Data Corresponding to Figure 4.  

For more detailed discussion on the LRS data and the data collection project which captured these data, please visit DDOT's GIS Wiki which discusses the data structure and provides additional background.

Extraction in Action

An intersection-leg model combined with cross-section linear event data provides us the ability to perform extremely detailed queries on our LRS.  Using the route/measure location of any intersection leg we can intersect with the cross-section data and extract a multitude of MIRE data.  Below is an expanded illustration of Figure 4 above, focusing on approach #4 on intersection A.  Using script tools, we can automatically access and extract all cross-section event data present at the beginning of the approach7.

Figure 5:  Example of Using an Intersection Leg (ID 4) to Access Cross-section LRS data for Route 456.

What can be extracted?

In just the Intersection Leg (Approach) category, we can use our Python script library to extract 12 out of approximately 40 MIRE elements in this category that would usually require a manual data collection:  For both states in the aforementioned MIS study, many of the elements in Table 3 were evaluated to be manual collections. 

MIRE Element (2.0)Value
Intersection Identifier for this ApproachA
Unique Approach Identifier4
Approach Mode1
Approach Directional Flow2
Number of Approach Through Lanes2
Left-Turn Lane Type2
Number of Exclusive Left-Turn Lanes1
Right-Turn Channelization1
Number of Exclusive Right-Turn Lanes0
Length of Exclusive Left-Turn Lanes30
Median Type at Intersection1
Approach Traffic Control1

Table 3: Example Results from Automated Extraction of Cross-section LRS Data.  

Conclusions, Next Steps

DDOT is convinced that the time and effort to collect high-value, 'core' roadway inventory data is a much better investment than capturing the MIRE data items individually.  Automating the intersection-leg model is easily done (we have referenced the script repository below).  An LRS roadway cross-section is not an insignificant investment, but one that has the potential to reward your agency many times over.  Once collected, we intend to incorporate cross-section events as a standard maintenance item when performing updates to our LRS.  This will ensure that items such as the 12 listed above will never need to be individually collected again; simply refreshing MIRE via scripts is now an achievable objective for DDOT.  Out of the 205 MIRE 2.0 elements which are not provided via HPMS we estimate that we can perform a fully automated extract for approximately 40% of those (81 total) via our Python script library for MIRE.  (Note:  This project is ongoing and this estimate is preliminary)  

One of the more exciting future possiblilities relates to what more we could do with our roadway cross-section data.  Once the MIRE Python automation scripts are complete, we hope to create additional Python scripts which can extract a routeable network from our LRS.  The elusive 'hard-to-get' piece has historically been turning movements at intersections.  Our LRS roadway cross-section data types contain the full array of turn lane types; after linking with our relational intersection-leg model, we believe we can use the lane type as a initial proxy to automatically generate a turning movements table for every intersection leg in the network.   

References

  1. HPMS State Panel Discussion, Highway Information Seminar, (2017)
  2. MIRE, Version 1.0, (2011) Page 1
  3. MIRE, Version 2.0, (2018)
  4. Background Report Guidance for Roadway Safety Data to Support the Highway Safety Improvement Program, (2011)
  5. "MIRE, MIRE, Pants on Fire", Presentation at the 2017 GIS-T, (2017)

  6. MIRE MIS Lead Agency Data Collection Report, (2013)
  7. DDOT Roads and Highways Tools Repository, (2018)
  8. Intersection Manager, Transcend Spatial Solutions, Inc (2018)
  9. Cross-section LRS Data, DDOT GIS Wiki, (2018)

text.