Scalable Bayesian Modeling, Monitoring, and Analysis of Dynamic Network Flow Data

ABSTRACT

Traffic flow count data in networks arise in many applications, such as automobile or aviation transportation, certain directed social network contexts, and Internet studies. Using an example of Internet browser traffic flow through site-segments of an international news website, we present Bayesian analyses of two linked classes of models which, in tandem, allow fast, scalable, and interpretable Bayesian inference. We first develop flexible state-space models for streaming count data, able to adaptively characterize and quantify network dynamics efficiently in real-time. We then use these models as emulators of more structured, time-varying gravity models that allow formal dissection of network dynamics. This yields interpretable inferences on traffic flow characteristics, and on dynamics in interactions among network nodes. Bayesian monitoring theory defines a strategy for sequential model assessment and adaptation in cases when network flow data deviate from model-based predictions. Exploratory and sequential monitoring analyses of evolving traffic on a network of web site-segments in e-commerce demonstrate the utility of this coupled Bayesian emulation approach to analysis of streaming network count data. Supplementary materials for this article are available online.

KEYWORDS:

• Bayesian model emulation
• Decouple/recouple
• Dynamic gravity model
• Dynamic network flow model
• Monitoring and anomaly detection

Supplementary Materials

The online supplementary files contain the appendices for the article.

Appendix A provides mathematical details regarding the gamma-beta dynamic discount model for time-varying Poisson rates that was described in Section 3.

Appendix B provides a schematic diagram that describes the process by which the Bayesian model monitors future predictions and adapts to deviations.

Acknowledgments

The authors thank Mark Lowe of MaxPoint Interactive for input throughout this project, and the editors and anonymous referees for their constructive input on our work that helped in revising the article. Kaoru’s research on this project was performed while he was a PhD student in Statistical Science at Duke University. Robert Haslinger research on this project was performed while he was with MaxPoint Interactive Inc., Morrisville, NC 27560.

Additional information

Funding

MaxPoint Interactive.