[{"slug":"interactive-recommendation","title":"Scaling LLM-based Recommendation","featured":true,"creationDate":"2024-12-20","category":"Project","excerpt":"Traditionally, recommendation systems have used techniques like collaborative filtering and content-based filtering. While these methods have been effective, the advent of Large Language Models (LLMs) has opened a new frontier allowing for highly tailored recommendation experiences. They also come with additional challenges, especially at scale. This project looks at the challenges of building LLM-augmented recommender systems.","labels":[{"id":"recsys","name":"RecSys"},{"id":"systemsengineering","name":"Systems Engineering"},{"id":"llm","name":"LLM"}],"cover":"interactive-recommendation.jpg","active":true,"draft":false,"html":"

info

This is an active project. It will grow with future updates, check back soon!\n

\n","readingTime":"1 min read","fullURL":"https://pingen.dev/projects/interactive-recommendation"},{"slug":"sustainable-computing","title":"Sustainable Computing","featured":true,"creationDate":"2022-09-03","category":"Project","excerpt":"The digital sector is expected to be responsible for 5% of global greenhouse gas emissions by 2025. Similarly it significantly impacts energy, water, and other natural resource consumption. And while large data centres near a 1.0 PUE and often run on green energy, indirect scope 3 emissions - for example in chip manufacturing - are difficult to monitor and therefore underreported. Moreover, these are not visible to end users, preventing them from making informed decisions.","labels":[{"id":"sustainability","name":"Sustainability"},{"id":"systemsengineering","name":"Systems Engineering"}],"cover":"sustainable-computing.jpg","code":"https://github.com/gpgn/sustainable-computing-sandbox","active":true,"draft":false,"html":"

info\n

This is an active project. It will grow with future updates, check back soon!\n

\n\n

Introduction

The average global temperature has been rising since the Industrial Revolution. Although there is some natural variability, there is predominant evidence indicating human activities are causing Earth to warm up, especially those that cause greenhouse gas (GHG) emissions (most importantly carbon dioxide, methane, and nitrous oxide). And while climate change is a known problem, atmospheric concentrations of this major driver have continued to increase in the last years.

Climate change impact includes extreme weather events; changes in sea level; droughts and fresh water scarcity; erosion and other soil degradation; dying coral reefs; frequent forest fires; and ecosystems becoming unhabitable for certain animal species. Even now, islands are dissapearing due to sea level rise. It is also a major threat to international peace and security, hightening competition for resources and leading to mass displacement and fueling socio-economic tension. Critically, the poorest countries face the greatest risks from these changes, although many of the worlds most climate-vulnerable countries are among the least polluting.

Many western countries and large companies are electrifying at a quick pace, and adding renewable energy sources into their energy mix. This however, is not without its issues, as for example The Netherlands is experiencing growing pains exemplified by major congestion of its power grid. These challenges are threatening timely climate action.

The digital sector is currently estimated to be responsible for 2-4% of global greenhouse gas emissions. This includes both production and consumption of digital technologies, for example computers, televisions, smartphones (production), and devices, data centres, and networks (consumption). This number is expected to grow to at least 6% by 2025, as data traffic is rising rapidly. Importantly, greenhouse gas emissions are just one example of the environmental impact of the tech secctor. Other major effects exist, such as depletion of energy, water, and natural resources.

$\"Figure$

Figure 1. Carbon emission scopes. Image taken from https://go.microsoft.com/fwlink/p/?linkid=2161861.

Sustainable Computing

The growth of the tech sector is unsustainable. However, changes can be made to turn that around. Industries are being established around recycling EV batteries. Companies are making steps to reduce their carbon footprint by transitioning to electricity from renewable energy sources. Additionally, the Power Usage Effectiveness (PUE) of large data centres is nearing 1.0 - meaning that the energy that is consumed by data centres is being used almost solely for compute (and not for cooling, and other supporting infrastructure).

These uplifting figures often end up in corporate sustainability reports, but only tell part of the story. Many companies report on Scope 1 and 2 emissions, but lack transparency on Scope 3 emissions as they are much harder to track.

As visualized in Figure 1, Scope 1 emissions are direct emissions from company-owned and controlled resources (small in the digital sector); Scope 2 emissions are indirect emissions from the generation of energy that is consumed by a company's resources (substantial in the digital sector); Scope 3 emissions are all indirect upstream and downstream emissions, e.g. those generated in the manufacturing, transport and recycling value chains.

As the carbon content of energy tends to zero, Scope 3 dominates carbon footprints. This means that much of the carbon impact of the tech sector is tied up in Scope 3 emissions. Think about the mining of raw minerals, shipping electronics around the world, and disposal of hardware. A recent Microsoft report estimated them to be up to 50 times the size of Scope 1 and 2 emissions (see Figure 2). Subsequently, usage of material resources are underreported as well.

The big cloud providers are slowly moving to expose Scope 3 carbon emission data. Microsoft and Google are currently providing detailed scope 3 data, but Amazon currently only does this under strict NDA. These figures however are very course-grained (yearly or monthly aggregations), rather than real-time and attributable to specific processes.

It is clear that we are moving in the right direction. However, two aspects that are severely lacking in this space are observability and awareness. While using renewable energy sources is a good step forward, it is important to think about the use of that energy. For example, how much energy or carbon emission is it worth to bump the accuracy of our cat detector model a little? And how are we able to measure that accurately and precisely?

$\"Figure$

Figure 2. Microsoft 2021 Scope 1, 2, and 3 emissions.

Measuring power consumption

There are currently few tools to estimate GHG emissions (especially Scope 3 emissions) in modern IT infrastructure stacks, which is typically deployed in shared cloud environments that have many processes making use of the same underlying hardware. Additionally, the IT services (compute, data storage, AI models, etc.) are being used by many different tenants. This makes it difficult to track and expose individual contributions to total carbon emission, and thus raise awareness or tie them into local carbon budgets.

A starting point for measuring carbon emissions is measuring power consumption (and multiplying that by carbon content). One reasonably accurate proxy for actual power readings is RAPL, or Running Average Power Limit. This is an interface that among other things exposes accumulated energy consuption of various system-on-chip (SoC) power domains. RAPL provides a set of counters providing energy and power consumption information. RAPL is not an analog power meter, but rather uses a software power model. This software power model estimates energy usage by using hardware performance counters and I/O models.

Thankfully, RAPL readings are highly correlated with actual plug power measurements. The processor has one or more so called packages. These are part of the actual processor. Each package contains multiple cores. Each core typically has hyperhreading, meaning it contains two logical CPUs. The uncore is the part of the package outside of the cores - including various components like L3 cache, memory controller, and possibly an integrated GPU. RAM is seperate from the processor. Depending on chip architecture, we have RAPL exposed for the full package (PKG); the cores (PP0), an uncore device which is usually the GPU (PP1), and the main memory (DRAM). See Figure 3 for a visual overview of the power domains.

In Linux, RAPL is exposed to user space through the power capping framework, and can typically be found under /sys/class/powercap/intel-rapl\\.

Note that in a machine, we might have other sources of energy consumption than the CPU, namely devices like GPUs; RAM; disk usage; the power supply itself; fans; cooling; and LEDs; although these typically consume (much) less energy.

To conclude, there are ways to measure power consumption. You can check this for yourself on your machine or laptop. However, things get a bit trickier in virtualized environments - for example Virtual Machines (VMs). These run on top of a hypervisor, and do not automatically have RAPL metrics exposed to them. If the hypervisor does not forward that information to the VM, there is no way to know. Unfortunatly, this is a typical situation in cloud environments, where you do not have access to the underlying host machine. And cloud providers have good reason not to expose that information directly.

$\"Figure$

Figure 3. CPU power domains supported by RAPL.

From power consumption to carbon emissions

Let's say despite all of these challenges, we obtain real-time process-level metrics on power consumption. We will also disregard Scope 3 emissions for the moment. How do we then determine the carbon content (Scope 2 emissions) of that used energy? We will recap a lot of information from Adrian Cockcroft's excellent slides from QCon London 2023. Let's take a look at the calculation for Scope 2 emissions:

Scope 2 Carbon = Energy Mix * PUE * Capacity Used * Emissions factor per capacity

PUE is not well standardized, but at least reported on for most cloud providers (and as mentioned, nearing 1.0). And based on the last paragraph, we assume we know the energy capacity and emissions factor on a fine-grained level (even user-specific, which is difficult to measure in a multi-tenant environment where infrastructure is shared).

Even including all these assumptions, we still run into some new challenges, for example: What is the energy mix in this specific location? Is it coal, hydro, solar, nuclear, ...? Do we have that data real-time (typically this is only known months later)? And do we have it at sufficient resolution (what if our process runs for a couple of minutes)?

The way cloud providers manage their power mix is through:

Standard utility contracts which are billed monthly (or delayed longer).
Power Purchase Agreements (PPAs) which are contracts to build and consume power generation capacity (solar, wind, etc.).
Renewable Energy Credits (RECs) which are purchases of renewable energy generation capacity on the open market from existing generators (which can also be claimed at a later time). There are different types of RECs (some of which do not even represent usable energy capacity, i.e. they are simply offsetting carbon emissions in another part of the world) which complicates things even more. They can also be traded for up to a year, which changes the entire calculation again, retroactively!

This makes it super challenging to determine the energy mix that was used at the time of a short-running process. We simply don't know exactly. Over time though, as the utility grid decarbonizes, this will be less of a problem. What's also great is that we are starting to see APIs that expose hourly grid mix data.

For now however, the main thing we can do is to use less energy. And for that, we need to empower the end-user by allowing them to make carbon-informed decisions.

Over time, we should move towards reporting real-time carbon metrics with open standards, to enable automated scheduling and optimization.

Tools and Ecosystems

Two main tools in the open-source cloud-native domain that focus on monitoring power consumption are Scaphandre and Kepler. Kepler and Scaphandre both want to give users an answer on how much energy spent by a workload (process, container, or VM).

Scaphandre does this by checking at every interval the PIDs of all relevant processes, and the RAPL output; and uses the CPU clock time/ticks taken from /proc/{pid}/stat and divides the total joules according to ratio. Kepler basically says this is not accurate enough because per-instruction power usage is different (so you cant simply divide according to ratio of scheduled time). Instead, it records the instructions per process (since each instruction uses different Wattage). It then estimates CPU time ratio based on ML model-based coefficients for these instructions.

Using these tools, and by integrating APIs like the Carbon Aware SDK - which can information on current and predicted GHG emmisions from energy consumption - we can start to build carbon-aware and even carbon-adaptive systems.

Kepler is part of a larger ecosystem of tools developed by contributors from Red Hat, IBM, and Intel. At this time development is still in very early stages though, and feels more like a proof of concept. Scaphandre on the other hand is developed by Hubblo and their set of open-source sustainability tools. Hubblo is a member of Boavizta, an international inter-organizational working group aimed at evaluating the environmental impact of digital technologies. In turn, Boavizta is a partner in the Sustainable Digital Infrastructure Alliance (SDIA).

Two other important groups doing interesting work in the domain are the Kubernetes TAG on Environmental Sustainability and the Green Software Foundation.

Experiments

As we've seen, there are still a lot of open challenges in this domain. Most notably, how can we accurately map power consumption to GHG emission and expose that to end users, to empower them to make informed decisions and enable effective regulation?

This is especially difficult for Scope 3 emissions, and involves many sub-questions. For example, how much Scope 3 emissions can be attributed to the use 100ms (or 3s, 10min, 24h, ...) usage of a given CPU chip (likely this will be tied to hardware depreciation expenses, as new hardware purchases embody new Scope 3 emissions). And subsequently, how can we precisely track that usage in a hyper-distributed, multi-owner, multi-tenant, and multi-modality cloud world?

The goal of this project is to experiment with aforementioned technologies, perhaps touching on a few of these challenges, and work towards an implementation that can be used in standard ML workflows. In the mean time, I hope to use the project to get more familiar with Rust.

info\n

This is an active project. It will grow with future updates, check back soon!\n

\n\n

References

[1] NASA.gov | World of Change: Global Temperatures

[2] Malmodin, Jens, et al. "ICT Sector Electricity Consumption and Greenhouse Gas Emissions–2020 Outcome." Available at SSRN 4424264 (2023).

[3] Khan, Kashif Nizam, et al. "RAPL in Action: Experiences in Using RAPL for Power measurements." ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS) 3.2 (2018): 1-26.

[4] Lipp, Moritz, et al. "PLATYPUS: Software-based power side-channel attacks on x86." 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.

\n","readingTime":"12 min read","fullURL":"https://pingen.dev/projects/sustainable-computing"},{"slug":"causal-discovery","title":"Causal Discovery","featured":true,"creationDate":"2022-08-11","category":"Project","excerpt":"Can we let autonomous systems discover generic rules that govern their environment on their own? Causal discovery is a method that tries to capture cause-effect relations from a mixture of observational and interventional data, but effectively generating new causal variables remains an open problem. This project looks at automatically learning high level causal variables from raw large-scale unstructured data.","labels":[{"id":"research","name":"Research"},{"id":"robotics","name":"Robotics"},{"id":"machinelearning","name":"Machine Learning"}],"cover":"causal-discovery.jpg","draft":false,"html":"

Introduction

Robotic platforms are employed for tasks that are too dangerous, costly, or simply too repetitive for humans to perform. A degree of autonomy is desirable to reduce the effort required from human operators to monitor and manage the system.

These autonomous systems can be deployed in constrained environments – as is the case for static robotic arms, or track-guided warehouse robots – or in unstructured open world environments which can contain unknowns. In the\nlatter case, an autonomous system will encounter changes in the environment, and possibly in the performance of its own capabilities. In order to still operate effectively and safely under these dynamics, it must have the ability to assess its own performance and adapt itself in response to changing circumstances. Such systems are referred to as Self-Adaptive\nSystems (SAS).

Self-adaptivity typically involves some form of feedback loop to allow the system to react on external stimuli. Many approaches to engineer SAS adhere to the processes defined in the MAPE-K control model: Monitor, Analyse, Plan, Execute, using shared Knowledge.

$\"Figure$

Figure 1. MAPE-K control loop.

The ability to model the environment under uncertainty is critical to successful autonomous operation. Recently, machine learning (ML) methods, and especially deep-learning based models, have been used to support or complement processes in the MAPE-K control loop, for example in data pre-processing, world modeling, or policy optimization.

However, while these models have shown great performance in tasks ranging from recommendation to visual object detection and language modeling, they are intrinsically obfuscated, and require another layer of models to provide global or local interpretability – spawning the field of Explainable AI (XAI). In addition, they have difficulty in generalizing to novel situations or tasks, and in reacting to shifts in data distribution. It is not straightforward to reuse learned representations.

Causal models, contrastingly, are less black-box, and by default more illustrative to a human operator. A causal model captures the causal mechanism underlying observational data in a Directed Acyclic Graph (DAG), with nodes representing the variables and directed edges representing causal properties such as direction, strength, and time-lag. Although causal models can be generated purely observational data, it is difficult to ascribe strong causal meaning due to possible (unobserved) confounding effects in the data. Many causal discovery methods based solely on observational data rely, therefore, on strong assumptions – i.e. on the existence of a set of variables that when measured remove bias introduced by confounders – or establish distinct bounds for the treatment effect.

If an autonomous system is able to actively intervene in a situation, possibly altering the sensor data distribution of a target variable, it can attempt to account for confounders. Classically, the Randomized Control Trial (RCT) is the golden standard for establishing causal relations. Through randomization, confounding effects on the target variable from any (un)known causes can be mitigated, ensuring that the intervention is the only cause of the observed effect.

$\"Figure$

Figure 2. Example of a DAG for a math camp example. Image taken from https://www.andrewheiss.com/blog/2020/02/25/closing-backdoors-dags/

Interventional data improves model robustness and generalizability compared to purely statistical methods such as ML. Real-world interventions, however, are typically expensive, and can even be infeasible operations in some domains. Simulations can circumvent these issues, but often struggle to realistically model the physical processes present in the real world, thus limiting their use. Therefore, one can combine observational and interventional data, thus compounding the statistical advantage of a large set of observational data with the causally stronger nature of interventions, if only a limited set.

Although there exists a plethora of studies into both self-adaptive systems and causal discovery, many studies consider only simulated data or systems. Even less investigate systems operating in real-life open-world scenarios. In this project in my time as a research scientist at TNO, some amazing colleagues and I set out to investigate the applicability of causal discovery methods for explainable self-assessment of autonomous systems operating in an open world, and integrated our solution into a continuous causal self-assessment pipeline on an operational autonomous system.

Specifically, we considered causal discovery methods for time-series data able to handle lagged correlations. Starting out with an initial causal structure derived from observational data, the system is able to introduce interventions to validate and update its internal causal world model. The goal here is not to establish a complete world model, or find the true causal model (if such a thing exists). As the system has a limited set of sensors and actuators, this is a futile task.

We are interested instead in producing a causal model that is robust to external changes and can generalize well to unseen environments, rather than one that captures world dynamics as accurately as possible. The model needs to be complete enough to generate effective adaptations against changes in the environment (reducing the number of human interventions), and decisions should sufficiently explainable to a human operator.

Background

We find many interpretations and frameworks of causality in literature, the most prominent of which are:

Pearl’s graphical causality, which is defined in terms of (perfect) interventions, by applying Structural Causal Models (SCMs) (or Structural Equation Models (SEMs)) and do-calculus.
Rubin’s Potential Outcomes framework which, while\nformally equivalent, does not deal with graphical representa-\ntions but instead compares different observed outcomes and\nrelates these to the causal model.
Wiener-Granger causality, which deals with time-series data, and only considers the predictive power of one time-series on the other - thus being an extremely weak notion of causality.

SCMs are a way of modeling causality that takes into account possible confounding effects and feedback loops. An SCM is a tuple $M := (I, J, X, E, f, P_{\\epsilon})$ with $I$ , a finite set of endogenous variables; $J$ , a finite set of exogenous variables; $X = \\Pi_{i\\in I} X_{i}$ , a product of standard measurable spaces; $E = \\Pi_{j\\in J} E_{j}$ , a product of standard measurable spaces; $f : X \\times E \\rightarrow X$ , the causal mechanism; and $P_{\\epsilon} = \\Pi_{j\\in J} P_{\\epsilon j}$ on $E$ , the exogenous distribution. This allows for a straightforward definition of interventions, as used in do-calculus. With $do(X_{I} = \\Xi_{I} )$ , the perfect intervention that sets $X_{I}$ to the value of $\\Xi_{I}$ , the SCM changes from $M = (X, E, f, P_{\\epsilon})$ to $M_{do(X_{I} = \\Xi_{I} )} = (X, E, f, P_{\\epsilon})$ with $f_{i} = {X_{I} = f(X, E) : i \\notin I} \\cup {X_{I} = \\Xi_{I} : i \\in I}$ , i.e. override specifically the causal mechanism that would set the values of the variables under intervention.

In order to reason about causality, especially when taking a statistical approach, several assumptions are made. The most prominent of which are the Causal Markov Condition, the Faithfulness assumption, and the Causal Sufficiency assumption.

The Causal Markov Condition states that all relevant probabilistic information that can be obtained from the system is contained in all its direct causes. In other words, conditional on the set of all its direct causes, a node in the causal graph is independent of all variables which are not direct causes or direct effects of that node. This assumption is used to reason causally about graphical models.
The Faithfulness assumption states that conditional independencies found in measurements do not arise from coincide, but from causal structure. In other words, if variables $X_{I}$ and $E_{I}$ are not correlated, they are really independent. This assumption is often made for constraint-based causal discovery methods.
The Causal Sufficiency assumption states that measured variables include all of the common causes (i.e. latent confounders are not present). This assumption simplifies the problem significantly, and is frequently made by score-based methods, but at the same time often violated in real-world scenarios.

\n\n

There exists much prior work on estimating causal structure based on purely observational data. These methods can roughly be categorized into constraint-based approaches, score-based approaches, and hybrid approaches.

Constraint-based methods, like the classic PC, IC, and FCI algorithms, test for conditional independence between variables, iteratively removing edges in the causal graph. Score-based methods, such as Bayesian EG or Adaptive Lasso, search for the DAG that best describes the statistical dependence relation in the observed data. Hybrid approaches combine both methods in order to reduce computational load, generally by pruning the search space used in score-based methods through conditional independence tests. The two main methods we considered in this project were PCMCI and TCDF.

PCMCI is a method for causal discovery specifically designed for large scale time-series data including both linear and non-linear (potentially time-lagged) dependencies. It is based on the classical PC algorithm but modifies it to a time-based variant by also taking into account data-points for a certain time-window in the past. This is the first step in the algorithm, but might lead to false positives in the case of taking especially liberal time-windows, or when time-series are highly interdependent or auto-correlated. Therefore in the second step, momentary conditional independence (MCI) testing, false positives are reduced by applying conditional independence tests including only the parent nodes in the conditioning set. Selecting an appropriate time-window therefore is important – which is a general notion when working with causal models and time-series data.

$\"Figure$

Figure 3. Causal discovery problem with time-series data. Image taken from Runge et al.

TCDF is a deep learning-based method for causal discovery. More specifically, it is based around Attention-based Convolutional Neural Networks (CNNs). To discover a causal structure in the supplied dataset, first, for each time-series in the dataset, a CNN is trained which aims to predict that time-series. All time-series in the dataset, including the target time-series, are provided as input to the CNN. After training, the internal parameters of the network are analyzed to determine which input time-series could be causes the of the target time-series for that CNN. In short, this analysis relies on the used attention mechanism, to determine which input time-series the CNN focused on for the prediction of the target time-series. Afterwards, all candidate cause-effects relations are validated using TCDF’s Permutation Importance Validation Method (PIVM), which measures how much the prediction error of a target time-series increases if the values of a potential cause are randomly permuted. This steps tests if the assumption of chronological ordering between cause and effect, i.e. cause must occur before effect, holds. In the last step the lag between cause and effect is determined through the analysis of kernel weights. All remaining cause-effect relations after validation are merged into a final causal graph.

$\"Figure$

Figure 4. Overview of TCDF framework. Image taken from Nauta et al.

Recently, JCI has been proposed as a framework for causal discovery on data found in multiple distinct contexts. Their approach pools data from multiple contexts in order to discover conditional independencies that can only be found when joining the measurements from both contexts, rather than only testing for conditional independence within each context and later combining those as done in most constraint-based methods.

Other work focuses on combining ML methods with causal learning, for example in order to learn causal variables (which are currently often designed, knowingly or unknowingly, by system engineers, or part of encoded neural-network representations); improving Reinforcement Learning robustness using causal world- or agent-models; and to understand biases in deep-learning models resulting from design decisions in data modeling or task selection through a causal lens.

System Overview

The main objective of the project was to investigate the effectiveness of current temporal causal discovery methods for application in real-world scenarios involving large-scale, uncertain and unstructured data for self-assessment of autonomous systems.

The ROS2-based system used in this work sits on top of Spot, the quadruped robotic platform designed by Boston Dynamics, and is comprised of several hard- and software components such as cameras, 3D sensing, object detection models, and audio components in order to communicate with humans. The Spot platform was selected for its agility and robust navigation of heterogeneous terrain types.

The system was tasked with a search-and-rescue mission to locate and assist a number of victims in an indoor and outdoor environment. Initialized with some prior knowledge about its environment, such as a rough map of the building, it may encounter unforeseen changes. Environmental dynamics include changes in lighting; decreased visibility due to fog or smoke; loud noise; blocked passages; or terrain variability (both in height variation and material density).

Use-case

The system was initially deployed in a hostile indoor environment, considered too dangerous for humans to enter. In this environment, it was tasked by a human operator to search for either a known number of humans believed present in the environment, or exhaustively search the space for an unknown number of victims. Victims can be of any gender or age, and may be positioned in varying orientations (i.e. standing, sitting, lying down) and be partially occluded from view. Additionally, they may be incapacitated, either physically or mentally, and may not respond to auditive queries by the system.

When a victim is found, the system will query them with a specific name in order to keep track of which victim is found. If the victim is unresponsive, and the system cannot assist, it will mark the victim’s location and delegate further responsibility to a human operator. Potentially, the system has to navigate in outdoor terrain as well when encountering blocked passages for example.

$\"Figure$

Figure 5. System architecture.

Figure 5 gives a high-level schematic overview of the components in the system.

After receiving a task (with optionally known locations of victims), the Tactical Planner component generates a flexible Markov Decision Process (MDP) from the concepts stored in the knowledge base. The MDP is then passed to a Monte-Carlo Tree-search based planner that is able to handle partially observable MDPs as well (where states are defined by distributions of possible actual states). The outcome of this process is an action, such as Go left, Go right, or Go to door. This is transferred via the Platform Server, which is responsible for operations relating to the robot’s navigation and other motor control (such as turning on a flashlight), to the navigational stack which interfaces with the Spot API and manages SLAM processes.

Microphone and audio devices are connected to the Audio Server which is used to parse speech signals into a textual representation, and parsed using an NLP method to localize victims in the vicinity of the system. In addition, multiple camera’s are used to generate imagery data, and to generate point clouds used to assess terrain state. Imagery data is used to detect victims using a task-specific object characterization algorithm. The Environmental Driver is used to interface with light, ambient noise, and gas sensors.

The Behaviour Server schedules the readings and commands to the four subsystems.

Finally, the Task Performance and Causal Reasoning components monitor the external and internal system’s state, and will generate an intervention when performance drops below the expected values required by the operator. The Causal Reasoning component specifically manages the causal world models established during operation.

The system is backed by ROS2’s DDS message bus implementation and a TypeDB graph database.

Causal reasoning

We'll zoom in a little on the causal reasoning component. It is initialized with a set of causal models generated previously using a causal discovery method (we can switch between methods, as long as it outputs a standardized DAG). On receiving a signal from the Task Performance component, and after an initial data filtering step to prune any missing values (which is common in real-world environments), the component will first determine which causal model is currently appropriate in the given context. This is done by calculating the Wasserstein distance between sensor distributions of the associated context of the causal model and the incoming data-vector (current environmental state). After selecting an active causal model, it will determine an appropriate intervention.

The intervention is generated by comparing the required (or demanded) performance for the current task as defined in the initial signal from the Task Performance component with the maximum expected effect obtained by applying an intervention as predicted by the causal model. The correlation of each intervention on the current target variable (relating to the task, i.e. detector frequency in the case of a visual search task) is determined by applying classic graph traversal and sums the contribution of all pathways by which the intervention and target variables are connected. The strength of every contributing pathway is determined by calculating the product of the path coefficients along that pathway. The resulting value together with the intervention (proposed behaviour) is returned to the Task Performance component when the expected value exceeds that of the signaled required/demanded value.

The generated causal models are stored and continuously updated on-the-fly with a causal discovery framework using a graph embedding approach. The full pipeline is specified in Figure 6.

All sensor components push their data onto the ROS2 DDS bus. This sensor data is first cleaned and aggregated in a pre-processing step, where missing values are filtered out, and sensor values generated in a single time step are grouped together. Next, causal models are created by sliding a time window over the data and generating causal models for every window. This results in a large dataset of graphs. Note that the specific causal discovery method is pluggable in this setup, and allows drop-in replacements. In the match & merge step, we prune that large set of causal graphs by using a graph embedding approach to determine significant differences between causal models by calculating relative distances in vector space. This results in a lower number of causal models that hold in the world in which the autonomous system operates. These can be viewed as the observed rules of the world that the system has learned.

Next, in the context association step, they are labelled with the distribution of sensor values under which the causal model was found (in order to later retrieve them). As described in the previous section, the system will select an active causal model when generating suitable interventions. The full process can be executed offline (which we did to create an base set of causal models), and then ran in a continuous online fashion in order to keep updating the system’s world model on the fly.

Causal models can also be merged into a single-world model when taking into account contextual variables by applying Mooij et al.’s work on causal models in multiple contexts. The proposed framework is designed to be lightweight and modular, in order to plug in new implementations in the future for the causal discovery; graph embedding; and context association steps.

$\"Figure$

Figure 6. Continuous causal reasoning framework.

Challenges & Lessons learned

With the previously described framework in place, we investigated the challenges associated with the application of causal discovery methods to two intervention scenario’s related to the visual search task.

First, the expected causal structure for both scenario’s were defined as a baseline. The first scenario involved various lighting conditions, for which it was expected that the image detector would fail to detect humans in certain conditions (e.g. strong back light, or dark conditions). The optimal intervention here is that the system can activate a flashlight in order to improve the lighting conditions and proceed with its task. As this is a relatively simple use-case, we expected that these causal relations would be captured by the causal discovery algorithm without major issues.

The second scenario involved varying terrain conditions (loose sand, grass, flat gravel, wood snips, obstacles, etc.). The expectation here is that terrain variability influences walking stability and speed, which in turn influences coverage. Additionally, there are several options to influence the systems walking behaviour, such as adjusting its walking height (i.e. low on the ground or high on its feet) or turning on a slow crawling walking mode. The system is more stable on rough terrain when positioning itself high on its legs due to the extra time it has to position its feet relative to the ground. When positioned low on its legs, this time is much shorter, leading to the robot being unstable and in extreme cases falling over. However, on flat terrain the low-positioned crawl mode is more stable and costs less effort.

Various combinations were recorded to be used as a dataset for terrain type. The full dataset includes point cloud data, local grid maps, efforts in joint states, and average speed. Additional metrics were calculated using the point cloud and local grid maps such as local omnivariance and surface variation based on work by Hackel et al..

After initial experimentation on small test cases taken from above-mentioned datasets, we found that the PCMCI method by Runge et al. and TCDF method by Nauta et al. were promising candidate causal discovery methods.

These methods are one of the few around that can handle large-scale time-series data, and have a solid implementation. TCDF, being a DNN-based method is more compute intensive however. Since the aim was to run all processes autonomously on the robotic platform, which is constrained for compute resources, we integrated the more lightweight candidate, PCMCI.

When doing these experiments, we identified a number of challenges thus far, which should be addressed in an operational framework:

Robustness. While there are many causal discovery methods available, there is a lack of results on real-life open-world data. Especially for autonomous systems, there is a need for a solid benchmark to indicate how well they transfer to novel situations, and whether they enable effective intervention selection in real-life scenarios.
Model accuracy. Our initial experiments still showed a significant degree of false positive relations, requiring significance thresholding to be effective. Causal graphs need to be compared to predefined true graphs for a given graph distance metric (i.e. edit distance, or more advanced methods with special attention for missed confounders).
Explainability. Causal models should be evaluated on their ability to explain the current state of the autonomous system to a human operator. Further, it would be interesting to expose examples from the match-and-merge step and allow an operator to select a model by hand based on those examples.
Bias. There exists an inherent bias in design/engineering of (autonomous) systems, simply by forcing a selection of variables for operational use through incorporating sensor components. This increases the chance of latent confounders.
Scalability. It should be clear what the performance impact is of incorporating additional signals into the framework, and what further steps can be taken to mitigate that impact (e.g. by more aggressive pruning in the merge step).

Future Work

Although there exists a lot of research on self-adaptive systems and causal discovery, most remains theoretical or validated on small simulated datasets. We found that causal models provide a straightforward and explainable way of reasoning about performance for an autonomous system, allowing the learned representation to be shared with other systems and reused for different tasks. However, some degree of feature engineering is currently still required in order to obtain useful variables for causal analysis, especially for dynamic environments. Causal Petri nets look like an interesting way forward.

Additionally, a system should be able to make an assessment of both risk and informational gain of a given intervention, for example by applying causal Bayesian networks. Second, the integration of expert knowledge and human feedback in a structured manner merits attention, especially because the graphical models lend themselves well to that application. Next, not only should the system be able to handle uncertainties in sensor data and its environment, it should also be able to manage uncertainties related to the goals and tasks it receives from human operators. Finally, as Schölkopf et al. noted, one of the most important challenges for real-world causal systems is the representation problem: i.e. to automatically learn high level causal variables from raw large-scale unstructured data.

Treading into causal territory should be done carefully. It is a continuous danger to blindly interpret graphical causal models as robust representation of causal reality, especially when generated and used by autonomous systems.

References

[1] Kephart, Jeffrey O., and David M. Chess. "The vision of autonomic computing." Computer 36.1 (2003): 41-50.

[2] Balke, Alexander, and Judea Pearl. "Bounds on treatment effects from studies with imperfect compliance." Journal of the American Statistical Association 92.439 (1997): 1171-1176.

[3] Pearl, Judea. "Models, reasoning and inference." Cambridge, UK: CambridgeUniversityPress 19.2 (2000).

[4] Rubin, Donald B. "Causal inference using potential outcomes: Design, modeling, decisions." Journal of the American Statistical Association 100.469 (2005): 322-331.

[5] Granger, Clive WJ. "Investigating causal relations by econometric models and cross-spectral methods." Econometrica: journal of the Econometric Society (1969): 424-438.

[6] Bongers, Stephan, et al. "Structural causal models: Cycles, marginalizations, exogenous reparametrizations and reductions." arXiv preprint arXiv 1611 (2016).

[7] Vowels, Matthew J., Necati Cihan Camgoz, and Richard Bowden. "D’ya like dags? a survey on structure learning and causal discovery." ACM Computing Surveys 55.4 (2022): 1-36.

[8] Spirtes, Peter, Clark N. Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.

[9] Elidan, Gal, and Stephen Gould. "Learning bounded treewidth Bayesian networks." Advances in neural information processing systems 21 (2008).

[10] Shojaie, Ali, and George Michailidis. "Penalized likelihood methods for estimation of sparse high-dimensional directed acyclic graphs." Biometrika 97.3 (2010): 519-538.

[11] Chickering, David Maxwell. "Optimal structure identification with greedy search." Journal of machine learning research 3.Nov (2002): 507-554.

[12] Wang, Yuhao, et al. "Causal discovery from incomplete data: a deep learning approach." arXiv preprint arXiv:2001.05343 (2020).

[13] Runge, Jakob. "Causal network reconstruction from time series: From theoretical assumptions to practical estimation." Chaos: An Interdisciplinary Journal of Nonlinear Science 28.7 (2018): 075310.

[14] Nauta, Meike, Doina Bucur, and Christin Seifert. "Causal discovery with attention-based convolutional neural networks." Machine Learning and Knowledge Extraction 1.1 (2019): 312-340.

[15] Mooij, Joris M., Sara Magliacane, and Tom Claassen. "Joint causal inference from multiple contexts." The Journal of Machine Learning Research 21.1 (2020): 3919-4026.

[16] Schölkopf, Bernhard, et al. "Toward causal representation learning." Proceedings of the IEEE 109.5 (2021): 612-634.

[17] Wright, Sewall. "The method of path coefficients." The annals of mathematical statistics 5.3 (1934): 161-215.

[18] Hackel, Timo, Jan D. Wegner, and Konrad Schindler. "Fast semantic segmentation of 3D point clouds with strongly varying density." ISPRS annals of the photogrammetry, remote sensing and spatial information sciences 3 (2016): 177-184.

[19] Anand, Ritwik, et al. "Petri Nets Enable Causal Reasoning in Dynamical Systems." A causal view on dynamical systems, NeurIPS 2022 workshop.

\n","readingTime":"24 min read","fullURL":"https://pingen.dev/projects/causal-discovery"},{"slug":"federated-cloud","title":"Federated Cloud","featured":true,"creationDate":"2022-08-10","category":"Project","excerpt":"As standardization of cloud native orchestration matures, the ease of creating and managing compute clusters similarly proliferates. Unfortunately, multi-cluster orchestration still has its issues, especially in resource-constrained edge environments. Can we build a robust interconnected fleet of mobile platforms that come and go at will, all sharing their compute resources and applications?","labels":[{"id":"research","name":"Research"},{"id":"systemsengineering","name":"Systems Engineering"},{"id":"robotics","name":"Robotics"}],"cover":"federated-cloud.jpg","draft":false,"html":"

Introduction

In the last 5-10 years, enabling technologies like Docker and Kubernetes and their respective open-source communities have driven the growth and maturity of cloud native development. The ease with which systems these days can be built to make use of cloud resources, and scale according to usage patterns, is incredibly empowering. This, in turn, has led to a boom in open-source software developed around open cloud native standards. Interfaces like the Container Runtime Interface (CRI), Container Network Interface (CNI), Container Storage Interface (CSI), and the Kubernetes API itself, allow developers to create complex applications agnostic to the underlying cloud vendor. And although the ecosystem and communities are maturing and big changes in the Kubernetes project are becoming sparse, the cloud native landscape is still vast and expanding.

$\"Figure$

Figure 1. Cloud native landscape. Image taken from https://landscape.cncf.io/.

Federated Cloud

As it has become relatively easy to create and manage Kubernetes clusters, organisations are growing them both in size and number. This phenomenon, known as kube-sprawl, is not just due to the relative effortlessness of cluster management. Often, organisations hit the scalability limits of a single cluster (at time of writing, Kubernetes recommends using no more than 5000 nodes in a cluster). Other times, they are forced to maintain multiple clusters due to organisational structure and security implications, geographical constraints (for example due to GDPR requirements), or explicit multi-cloud strategies to reduce vendor lock-in.

These are all solid reasons to want to move towards a multi-cluster setup. However, multi-cluster is also expensive, as compute resources often sit idle and clusters need to be managed separately.

Federated cloud is aimed at joining multiple clusters together to form a single interconnected cloud (at least, from a user's point of view), allowing users to share compute resources and move workloads around seamlessly. In this way, workload scheduling can be further optimized, and both data and applications can be migrated in a standardized cloud native way.

However, federated cloud also poses technical challenges that are not solved by Kubernetes itself (Kubernetes is currently scoped to the single cluster boundary, although with the advent of multi-cluster services that is slightly changing). The main ones are:

Inter-cluster networking. The problem of establishing network connection between heterogeneous clusters (each with a different networking and cluster configuration) to allow inter-cluster Pod-to-Pod and Service-to-Pod routing.
Multi-cluster deployment orchestration. The problem of providing a single control plane and resource model to deploy and manage your applications across clusters in a Kubernetes-native way.
Multi-cluster data management. The problem of managing stateful workloads by providing data locality and robust native migration of different data sources and formats.
Multi-cluster security models. The problem of dynamically restricting access to applications, data, and compute resources in a multi-tenant scenario to the agreed upon scope.

Multi-cluster topologies

There are different models for establishing a federated cloud. The simplest model is a hierarchical one, where a single cluster is used as a host cluster, and other clusters are able to join the federation. In this setup, the host cluster serves as the control plane for the federation. Applications are scheduled to the host cluster, which determines the best placement, and pushes the request to one of the joined clusters. A tiered waterfall structure can also be created where a downstream cluster in turn acts as a host cluster for other clusters and delegates any incoming deployment requests.

An alternative is a decentralized model, where no single host cluster is identified. Rather, peer-to-peer connections are established between clusters, and any cluster is able to delegate deployments to another. Note that this does not have to mean that clusters are fully interconnected, or that peering is always bi-directional. A cluster can request compute resources in another cluster, but does not necessarily have to accomodate for that itself. This model is significantly more flexible, as we can have clusters that are able to make use of another cluster's resources in an ad-hoc fashion, even though it might not be able to share resources itself. Moreover, we no longer have a single point of failure in the central host cluster. It also introduces additional complexity however, for example in the case of loops, additional communication overhead, and elaborate multi-cluster topologies.

Another way is a hybrid form, where part of the federated cloud is applying a centralized model, and other parts are set up in a decentralized manner.

$\"Figure$

Figure 2. Multi-cluster topologies. Images taken from https://admiralty.io/docs/concepts/topologies/.

Technologies

There are a number of projects within the cloud native landscape that tackle the above-mentioned challenges in the federated cloud space. Some focus on establishing inter-cluster network connection, some aim at creating a better deployment model and multi-cluster control plane, and some try to solve both.

Specific challenges in the multi-cloud networking domain are inter-cluster service discovery, inter-cluster service invocation, and inter-cluster network policies. There are solutions that are baked into the container network interface - for example Cilium ClusterMesh - or in a service mesh - for example Istio, that routes it traffic through sidecar proxies. Other solutions are deployed seperately. Submariner is a well-known project that provides cross-cluster L3 connectivity and multi-cluster service discovery through a broker architecture.

Considering workload orchestration, Kubefed was a Kubernetes subproject aimed at providing a federated deployment model The downside was that it did not provide a way to work with Kubernetes-native resources, and that it only supported a centralized topology. It is now discontinued but has a successor in the novel Work API proposal. Other solutions support decentralized (peer-to-peer) topologies, such as Admiralty and Liqo. The Liqo paper goes into extensive detail about their approach.

GAIA-X

One special federated cloud project is the GAIA-X project. GAIA-X is a European initiative, aimed at establishing a secure and federated data infrastructure. Building an open federated cloud that "stands for European values", it should provide digital sovereignty to citizens. The idea is to create an alternative to the large US-based hyperscalers (Amazon, Google, Microsoft), by allowing European cloud providers to inter-connect their cloud infrastructure and data platforms through common standards. End-users would then be able to run their applications on top, without having to worry about which cloud their software would end up on. They would just provide their specification (i.e. performance and network constraints, geographical constraints, cost constraints, carbon emission constraints, etc.). It is still uncertain if the project will be a success, but the aim is admirable.

$\"Figure$

Figure 3. Liqo multi-cluster model. Liqo employs the Virtual Kubelet project to represent other clusters as a big node within a cluster, allowing for native scheduling. Liqo also sports a nifty data fabric feature that simplifies stateful workload orchestration.

Edge environments

Now that we have learned about federated cloud, we would like to start applying it to our systems. Specifically, it would be great to be able to apply these technologies to edge platforms such as UAVs. That way, we could seamlessly connect fleets of drones that can share compute capacity or machine learning models, for example for wildlife or wildfire surveillance, or to establish an ad-hoc network in the case of environmental disasters (e.g. the recent floodings in north-west Europe).

Unfortuntely, most of the aforementioned solutions assume a static and high-bandwidth environment. They are not developed for constrained and dynamic edge environments where we have clusters joining and leaving on the fly. More, the network can be flaky and drop completely at times. This is not an ideal environment for standard solutions. There are a few technologies that facilitate edge orchestration in resource-constrained environments (such as KubeEdge, FLEDGE, or KaiS), but those do not support multiple clusters.

Security

Hardening a single Kubernetes cluster is important, and the project has - especially in these last few years - made significant steps in improving its security methods (we could be close to effortly running core Kubernetes components rootless 🎉). The NSA has published a report with some sane default steps to improve cluster security.

Full cluster separation is obviously the safest, and a direction mamy companies are now taking (hence the kube-sprawl). However, what if we do want to share some of our resources and applications with others? Kubernetes has the concept of namespaces, but those provide only very weak isolation.

Allowing other tenants to run workloads on your own cluster introduces many additional security considerations. Tenants should be restricted to the resources they are granted, and not see any of your own compute resources, data, and applications, nor any of the other tenants'. Additionally, you should not be able to see the data and applications of the tenants using your cluster.

One project that allows multiple tenants to share a single cluster is the vcluster project by Loft. Tenant Pods end up on the same shared infrastructure, but tenants can only view and access their own virtual cluster (control plane included). Vcluster employs Pod Security Standards, as well as Network Policies and other measures such as enforcing rootless containers to secure tenant workloads.

Further ways of hardening involve employing container runtimes with an improved security model, such as Kata Containers, although adding additional security measures typically comes at a performance cost and does not necessarily reduce the attack surface. Container runtimes that advertise themselves as more secure, often expose a much wider interface to the host and invoke more calls to the host kernel. They also introduces a performance tradeoff that is especially important in resource-constrained edge environments.

Experimentation

To experiment with secure federated cloud technology in edge environments, we built a local test bed. Through a mix of Vagrant, libvirt, Terraform, and kubeadm, we could run multiple clusters on our local workstations. That sped up development significantly.

To simulate reduced network performance, we used Chaos Mesh, a chaos engineering tool that is able to inject faults into your system. These could be application or Kubernetes component failures, or network issues such as increased latency, packet loss, and reduced bandwidth. Under the hood it uses tc to apply these rules on your cluster hosts. At the time Chaos Mesh did not support multi-cluster functionality, so this was custom built. With this setup we could run specific edge scenario's in a reproducible manner.

We used Liqo as the federated orchestration solution, providing both a Wireguard-tunnel-based peer-to-peer network solution, as well as a simple Kubernetes-native deployment model.

Next, we needed to be able to react to changes in our cloud environment. The default Kubernetes scheduler does not take into account network conditions when deciding on wich node Pods should land. This severely impacts performance for Pods with specific network requirements requirements. We integrated Intel's Telemetry Aware Scheduling (TAS) solution to add custom scheduling rules that take into consideration cluster-to-cluster network latency, packet loss, and bandwidth. To report these metrics to the TAS scheduler, we need to observe them. This was done through OLSR, a routing protocol optimized for mobile ad-hoc networks that exposes network quality information.

Conclusion and Future work

Federated cloud technology is still in its infancy, but it has great potential to further push the interoperability and ubiquity of cloud computing, enabling seamless processing capabilities at the edges of our world, and perhaps beyond.

Its remaining challenges - including those in edge environments - are rapidly being addressed by both academia and industry, and it's exciting to see what's next. These are just a few examples of inspiring future directions:

Fleet management: There are currently few projects that address the UI/UX aspect of multi-cluster, multi-device, and really multi-anything environments. Kubernetes is a great platform to build platforms, and more and more solutions are popping up to manage not just compute resources, but even physical objects. We really need a better standardized management interface for that purpose.
Orbital Edge Computing: Federated cloud, moving towards robust edge application, is an ideal enabler for future low-earth-orbit (nano-)satellite communication. Specific mention to Denby et al., where I first saw the term.
Sustainable Computing: Federated clouds, such as those envisioned by the GAIA-X project, would open the market and increase competition on environmentally oriented performance signals, such as carbon emission. It could allow end users to easily choose the most environmentally friendly cloud to run their workloads on. If you're interested in this, be sure to check out one of my other projects.

References

[1] Arbezzano, G., and Palesandro, A. "Simplifying multi-clusters in Kubernetes." Cloud Native Computing Foundation - Blog

[2] Iorio, Marco, et al. "Computing without borders: The way towards liquid computing." IEEE Transactions on Cloud Computing (2022).

[3] Fogli, Mattia, et al. "Performance evaluation of kubernetes distributions (k8s, k3s, kubeedge) in an adaptive and federated cloud infrastructure for disadvantaged tactical networks." 2021 International Conference on Military Communication and Information Systems (ICMCIS). IEEE, 2021.

[4] Goethals, T., De Turck, F., & Volckaert, B. (2020). Extending kubernetes clusters to low-resource edge devices using virtual kubelets. IEEE Transactions on Cloud Computing, 10(4), 2623-2636.

[5] Han, Yiwen, et al. "Tailored learning-based scheduling for kubernetes-oriented edge-cloud system." IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 2021.

[6] van Rijn, Vincent, and Jan S. Rellermeyer. "A fresh look at the architecture and performance of contemporary isolation platforms." Proceedings of the 22nd International Middleware Conference. 2021.

[7] Denby, Bradley, and Brandon Lucia. "Orbital edge computing: Nanosatellite constellations as a new class of computer system." Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 2020.

\n","readingTime":"13 min read","fullURL":"https://pingen.dev/projects/federated-cloud"},{"slug":"hottarget","title":"Hot Target Detection","featured":false,"creationDate":"2016-10-02","category":"Project","excerpt":"Hot targets such as wildfires and volcanoes have a devastating impact on the planet, our infrastructure, and health. Detecting hot targets early and monitoring their progress is important to successfully manage and control them. Studies have shown that wildfires can be detected in Landsat 8 imagery by using Top of Atmosphere (TOA) reflectance, but false alarms are frequent. In this project, I took a deep neural network approach to hot target detection in satellite imagery.","labels":[{"id":"research","name":"Research"},{"id":"machinelearning","name":"Machine Learning"},{"id":"geo","name":"Geo"},{"id":"ipcv","name":"Image Processing"}],"cover":"hottarget.jpg","draft":false,"html":"

Introduction

Hot targets such as wildfires and volcanoes have a devastating impact on the planet, our infrastructure, and our personal health. Wildfires across the globe result in a yearly additional $3.5 \\cdot 10^{15}$ g to the existing atmospheric carbon emission.

If the aim is to decrease emission numbers, grassland, savannah, and deforestation fires are the prime targets for hot target detection. Fire emission time series analysis conducted by van der Werf et al. shows that the largest contributors to carbon emissions were grassland and savannah fires (44%), with tropical deforestation and degradation fires (20%), woodland fires (16%), and forest fires (15%) being big factors as well.

Most fires in remote areas and at higher elevation are caused by lightning, while we see an inverse effect near urban conglomerations. Wildfire detection is increasingly important, due to the globally increasing prevalence of wildfires. Westerling et al. show that wildfires have increased in both frequency and duration in the US since the 1980’s. We also see a global increase in yearly burned area, though there is quite some spatial variability in these assessments.

However, the risk and hazards of wildfires are not restricted to just atmospheric carbon emission. Humans, wildlife, and vegetation are severely affected as well. Injury and death resulting from exposure to heat and smoke inhalation, and trauma due to the loss of structural integrity are just two of the most prominent examples of the adverse effects of wildfires for humans. Other examples include indirect exposure to chemicals released during a fire, through water or soil contamination.

The consequences of the destructive nature of wildfires detailed above take years to restore, and can cost billions: the US Forest Service estimates the cost of fire suppression to increase to nearly USD 1.8 billion by 2025. That is leaving out other costs such as ecological and infrastructural reconstruction, and medical aid.

Clearly, effective detection, suppression, and prevention are necessary to combat wildfires. Detecting hot targets early and monitoring their progress is important to successfully manage and control these situations.

\n\n

Detection and suppression of both urban and rural wildfires has been important for many civilizations throughout history. In ancient Rome, the Vigiles was a proto-firefighting brigade created by Nero (though they could not prevent the Great Fire of Rome), and similar watch forces were maintained in Europe through the ages. More modern approaches to fire detecting came in the 20th century, when fire lookout towers, infrared camera’s, and smoke detectors where employed. These approaches are referred to as remote sensing, which is the collecting and interpreting of information about the environment and earth’s surface without making physical contact. Fire analysis using satellite imagery or aerial data, also part of remote sensing, was first performed in the 1980’s. Beside local sensor networks and regular fire lookout towers, it is one of the main practices for fire detection used today.

Following, a brief overview of relevant research in automatic hot target detection using satellite imagery. We will also expand on relevant research in machine learning, and its role in hot target detection.

Remote sensing

Remote sensing approaches such as wireless sensor networks are growing in terms of research and implementation. Wireless sensor networks\nhave been shown to be effective in detecting and forecasting forest fires\nin real-time, as opposed to satellite-imagery-based methods that have low\nspatial and temporal resolution. Yu et al., for example, propose a neural network paradigm for wireless sensor networks in which sensor nodes collect data (i.e. temperature, humidity, smoke) that gets sent to a cluster node that - together with other cluster nodes - processes the data using a neural network. The neural network takes the input data and produces a weather index (likelihood for the current weather to cause a fire) and reports it to a manager node, which in turn produces a fire danger rate. Though no comparison with imagery-based methods is made, their neural network approach is more efficient than other in-network processing methods.

Most existing data-driven approaches of fire detection however, are satellite-imagery-based. The moderate-resolution imaging spectroradiometer (MODIS) was launched into orbit aboard the Terra (1999) and Aqua (2002) satellites respectively. Combined, Terra MODIS and Aqua MODIS can map the Earth’s surface in 1 to 2 days, obtaining data from 36 spectral bands. These bands come in 3 spatial resolutions: 2 bands at 250m/px, 5 bands at 500m/px and 29 bands at 1km/px. MODIS produces global fire products every day using the original detection algorithm by Kaufman et al. and currently the improved contextual algorithm proposed by Giglio et al.. This implementation of the MODIS fire detection algorithm relies - besides pre-/post-processing steps like cloud masking and sun-glint rejection - on manually selected thresholds for top of atmosphere (TOA) reflectance/radiance, though improvements are still being actively researched.

Fire detection algorithms based on data acquired by the Visual Infrared Imaging Radiometer Suite (VIIRS), an imager with a higher spatial resolution than MODIS, also rely on these manually selected thresholds. VIIRS launched in 2011 aboard the Suomi-NPP satellite, and obtains spectral imaging data from 21 bands (16 at 750m/px and 5 at 375m/px). A second VIIRS is expected to launch aboard the JPSS-1 in 2017 on the same orbit as the first VIIRS. VIIRS fire products are generated using a stripped down version of the MODIS algorithm, where C4 (The 4th iteration of the algorithm) is used for the 750m/px product, and C6 for both 750m/px and 375m/px products. These fire products are used for further analysis on, for example burned area mapping, modelling of freight traffic, or combustion source characterization.

$\"Figure$

Figure 1. Bandpass wavelengths for the Landsat 8 OLI and TIRS sensors, compared to the Sentinel-2 MSI sensor, and Landsat 7 ETM+ sensor. Image obtained from https://landsat.gsfc.nasa.gov/article/sentinel-2a-launches-our-compliments-our-complements/.

Even higher resolution imagery is provided by the Landsat 8 satellite. Landsat is the longest running satellite imagery program, running since 1972. The latest satellite, Landsat 8, was launched in 2013 and provides images with a spatial resolution of 15 to 100 m/px and a temporal resolution of 10-16 days. Landsat 8’s Operational Land Imager (OLI) can acquire data in 9 spectral bands with 30m/px spatial resolution, while the Thermal Infrared Sensor (TIRS) collects 2 spectral bands with 100m/px spatial resolution. Figure 1 shows the bandpass wavelengths for the Landsat 8 sensors. High spatial resolution imaging data acquired by the Landsat satellites have been used for a range of topics, including hot target detection such as volcanism and fires. Due to open access policies, the full Landsat archive has been made publicly available by NASA/USGS. The European Space Agency (ESA) has also adopted similar policies, providing access to data obtained through their Copernicus program (including the latest Sentinel-2 missions), as well as Japan and their ASTER mission. However, even Sentinel-2 data ranging up to 10m/px is insufficient to chart the small fields in the Bangladesh delta, exposing the need for non-satellite based methods as well.

Fusion of different imaging data has also been used to overcome the spatial and temporal limitations of each imager. Boschetti et al. have used MODIS-Landsat fusion to identify burned area with MODIS active fire detection at a 30m/px spatial resolution. Similar fusion has also been applied for gap filling, and radiometric normalization. Murphy et al. present a novel global hot target detection algorithm (HOTMAP) with high detection rates (80%) and low false positive rates (<10%) that incorporates both Landsat 8 and Sentinel-2 data. An visualization of an implementation of their novel daytime detection algorithm can be found in Figure 2. Fusion of Landsat and MERIS imagery has been performed by Zurita-Milla et al. by applying unmixing-based data fusion to combine Landsat’s spatial resolution with MERIS’ spectral resolution.

$\"Figure$

Figure 2. Classification of hot pixels in Landsat 8 data in the vicinity of Adelaide, Australia (LC80970842015004LGN00). a) Landsat 8 spectral bands 2 (top left); 5 (top right); 6 (bottom left); and 7 (bottom right). b) False colour RGB image of bands 2, 3, and 4. c) Binary image of hot pixel classification output by the algorithm proposed by Murphy et al. d) Hot pixels marked in red superimposed on the original false colour RGB image.

Learning-based remote sensing

There have been a number of approaches to remote sensing that rely on machine learning. Petropoulos et al. have investigated the use of SVMs with Landsat data to perform burned area mapping with high accuracy. Others use SVMs to classify urban areas in an active and semi-supervised learning context, in which the SVM is fed training data, annotated by a human expert, that is expected to be most effective for training. A full review of the use of SVMs in remote sensing is provided by Mountrakis et al..

In the domain of remote sensing, some research on DNNs has been done. Basu et al. use a DNN approach to classify satellite imagery in land cover classes, and obtain impressive results on the SAT-4 and SAT-6 datasets, outperforming other deep learning methods. Castelluccio et al., and Hu et al. utilize a CNN to classify different types of land (i.e. forest, buildings, beach) with great success.

Data

we use multispectral Landsat 8 imaging data obtained from open-access portals, such as Amazon AWS. Data from these portals can be used in combination with search portals such as the USGS Earth Explorer and news sources to easily locate and obtain imaging data from large and small wildfires. We use all multispectral data as input for our machine learning methods, except OLI band 8, since the spectral information in this panchromatic band is already captured by bands 2, 3, and 4 (see also Figure 1). Although the Near Infrared (NIR) and Short-wave Infrared (SWIR) bands are commonly used in hot target detection, the other bands, including the Thermal Infrared Sensor (TIRS) bands, can provide additional information. The machine learning methods may pick up on more subtle relations between irradiative energy output on different wavelengths.

TOA reflectances (corrected for solar angle using Equation 1), and TOA radiance from all used bands will be used as input for our network, including an 8-neighbourhood window. For the 11-band Landsat 8 scenes, excluding OLI band 8, this would result in an input layer of 10x3x3 neurons.

$\n\\begin{equation}\n\\begin{align}\n \\rho \\lambda = \\frac{\\rho \\lambda '}{\\cos(\\theta_{SZ})} = \\frac{\\rho \\lambda '}{\\cos(\\theta_{SE})} \\text{, with} \\nonumber \\newline\n\\rho \\lambda = \\text{TOA planetary reflectance} \\nonumber \\newline\n\\theta_{SE} = \\text{Local sun elevation angle} \\nonumber \\newline\n\\theta_{SZ} = \\text{Local solar zenith angle} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

However, to be able to use our machine learning models, we need annotated data. Meaning that beside Landsat or Sentinel scenes, we require the exact positions of hot target pixels in these scenes to be able to allow the network to learn which pixels are hot targets. The size of academic literature on hot target detection algorithms is relatively small, and annotated datasets hard to come by. This makes evaluation of any new method quite labour intensive and error-prone. We augment our data to simulate hot target pixels to generate training data for our network, by extracting obvious hot target pixels from Landsat scenes and incorporating these features in a new dataset. This method however, might cause the machine learning methods to overfit, and we lose the generalization property of machine learning. We have therefore reached out to the authors of the HOTMAP algorithm with the request of sharing their evaluation set consisting of 45 Landsat 8 OLI scenes, manually annotated by human analysis. They have kindly allowed us to use this dataset for our research.

In this dataset, there are five Landsat 8 scenes per geographic region, including radiance and TOA reflectance maps, and pixel coordinates of hot targets. We use this dataset as an evaluation benchmark, together with the simulation dataset. From here on out, we will refer to this dataset as FIRES. An example scene of the FIRES dataset is provided in Figure 3.

$\"Figure$

Figure 3. Example data of the FIRES dataset by Murphy et al. used in this research. Imagery was taken in the vicinity of Alaska, USA (LC80690182014140LGN00). Shown are OLI bands 1 to 9 (excluding Panchromatic band 8), TIRS bands 10 and 11, a False Colour (FC) composite image, and the binary Ground Truth (GT) labelling.

Methods

We will look at 3 different methods, HOTMAP (highlighted in previous sections), an SVM-based approach, and a DNN-based approach. Our hot target detection method is compared to the findings of Murphy et al.(HOTMAP) and Giglio et al. (CA).

HOTMAP

HOTMAP, the algorithm proposed by Murphy et al., depends on a number of parameters, mainly the $\\alpha$ and $\\beta$ parameters. They can be described as boolean mapping functions, taking a multispectral image (converted to TOA reflectances) as input and outputting binary maps. Their formulas are shown in Equations 2 and 3.

$\n\\begin{equation}\n\\begin{align}\n \\alpha = \\frac{\\rho_{7}}{\\rho_{6}} \\ge 1.4 \\text{ AND } \\frac{\\rho_{7}}{\\rho_{5}} \\ge 1.4 \\text{ AND } \\rho_{7} \\ge 0.15 \\text{, with} \\nonumber \\newline\n\\rho_{n} = \\text{TOA reflectance in band n} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

$\n\\begin{equation}\n\\begin{align}\n \\beta = (\\frac{\\rho_{6}}{\\rho_{5}} \\ge 2.0 \\text{ AND } \\rho_{6} \\ge 0.5) \\text{ OR } S_{6} \\text{ OR } S_{7} \\text{, with} \\nonumber \\newline\n\\rho_{n} = \\text{TOA reflectance in band n} \\nonumber \\newline\nS_{n} = \\text{TOA reflectance is saturated in band n} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

The $\\alpha$ and $\\beta$ maps are merged using a simple logical OR operator ( $\\alpha$ OR $\\beta$ ), after which a clustering algorithm generates clusters of hot target candidates. Clusters that do not contain at least a single $\\alpha$ pixel are discarded. Remaining clusters are labelled as hot targets.

The $\\alpha$ parameter is designed specifically to produce as little false alarms as possible, but to detect at least a single pixel in a fire cluster. Its precision means that it can sometimes miss hot target clusters (especially on saturation of band 7 with very hot targets), which is why the $\\beta$ parameter is used in combination with the $\\alpha$ parameter. The $\\beta$ parameter is used to detect the particularly hot targets, for example in cases of saturation of bands 6 and 7. The $\\beta$ map will have good accuracy, but can also produce a large number of false alarms. When used together, the parameters provide effective hot target detection.

We implemented the algorithm to verify its effectiveness on the\ndataset. We will compare our results to the results reported in [81] consid-\nering we use the FIRES dataset as well.

SVM

We also implemented an support vector approach for hot target classification. We initialize our classifier with a radial basis function kernel, with kernel coefficient $\\gamma = 1$ n with $n$ = number of features, in accordance with our feature vector size. The other hyperparameters were tweaked as part of our experiment. Each pixel is used as a data point, consisting of 10 features, being the combined OLI and TIRS channels. The binary labels are used for ground truth validation. To ensure a balanced dataset, we use all hot target data points, and use the same amount of non-hot target data points. The rest of the non-hot target data points are discarded. We randomize the data before we apply pruning. This results in a total size of 43546 data points (with 21773 hot target pixels). Using 10-fold cross-validation we utilize the entire dataset, each fold being employed as validation set once.

DNN

We use the same architecture as in the Ground Cover Analysis project, but adapt it to be applicable for multispectral images. Instead of a 3-channel 8-bit RGB input, we use 10-channel 16-bit multispectral images. The output (and concurrently validation data) remains the same, a binary segmentation map the size of the original input image. We again employ data augmentation (rotations, and elastic transformations) to generate more training data, and so allow the network to encounter a broader range of data. We also train an MLP using the previously mentioned feature vectors, with Adam stochastic gradient based weight initialization. The MLP’s hyperparameters were tweaked as part of the experiments.

Results

To benchmark our deep neural network implementation of hot target detec-\ntion we implemented the CA, and HOTMAP algorithms. In addition, we implemented an SVM and a simple MLP neural network. We will start with an overview of the accuracy and precision scores, and continue with\na more expansive view of our results - most notably the machine learning\nresults.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Method	Accuracy	Precision
CA	0.9999	0.2397
HOTMAP	0.9999	0.6766
SVM	0.9847	0.9954
MLP	0.9844	0.9951
DNN	0.9797	0.0015

Table 1. Hot target detection by method.

CA & HOTMAP

The contextual algorithm (CA) of Giglio et al., and the HOTMAP algorithm both provide excellent accuracy. However, this good result is primarily due to the large number of non-hot target pixels in the satellite imagery. We get a clearer picture of how these algorithms perform when looking at the precision scores, the fraction of true hot targets over all marked hot targets (also called positive detection rate in Giglio et al.). In line with Murphy et al., we observe that CA results in many false positives, while HOTMAP performs much better. We believe this difference is caused by incorporating extra information from additional wavelengths.

SVM & MLP

The SVM and MLP were both trained on a dataset consisting of 50% hot target pixels and 50% non-hot target pixels (selected randomly from the complete FIRES dataset). We obtain exceptional precision scores, even with 10-fold cross validation. Accuracy scores are slightly lower than CA and HOTMAP, but this is due to the difference in dataset size. We ran CA and HOTMAP on the full images, the majority of which were non-hot target pixels, while the SVM and MLP saw only 50% non-hot target pixels. When we use the trained SVM to predict on full images, we achieve similar accuracy results.

To determine the effectiveness of these methods we first determined the data separability in n-dimensional feature space, by generating crossplots of the pixel values in every wavelength band, and labelling them with our ground truth (hot target or non-hot target). We observed that there exists a near linear separability when comparing the values of OLI band 6 versus that of OLI band 7.

We also looked at the effect of our features (being the 10 OLI and TIRS channels) on the performance of the SVM, by removing these features from our dataset. In line with the results of our visual inspection of separability, we observe that removing the information in OLI band 7 results in the most severe performance drop.

DNN

The DNN fails to perform with any precision, as the results in Table 1 clearly show. When we investigate individual segmentations, a batch of which is displayed in Figure 4, we see that not only does the DNN not pick up the hot targets, it also tends to label other structures such as cloud formations as hot targets.

This detrimental effect could be caused by the lack of large swaths of hot targets. The hot targets in our satellite imagery are usually only a couple of pixels large, while our DNN architecture works better for larger targets, as is evident by our ground cover analysis results. We tried a number of changes in architecture, similar to those reported in previous ground cover analysis sections, but found no major improvement.

$\"Figure$

Figure 4. Examples of DNN segmentations of augmented satellite imagery. Shown are the original OLI band 7 image (left), ground truth hot target annotation (middle), and DNN labelling (right).

Concluding

Existing algorithms of Giglio et al. and Murphy et al. both have excellent accuracy. This is due to the combination of the algorithms effective ability to discern non-hot target pixels (under-classification) and the large number of non-hot target pixels in our FIRES dataset. When we consider precision however, the fraction of accurately labelled hot targets out of all labelled hot targets, we see that there is definitely room for improvement.

The existing algorithms were able to be run on the original input image, and somewhat resemble the colour indices methods of ground cover analysis. Their advantages, much like the colour indices methods, are the speed of execution, simplicity, and ability to handle large input. Their disadvantages are immediately apparent in the precision scores listed in Table 1.

When we look at our machine learning methods, we see that these methods can obtain a much higher precision rate. When we investigate the feature space, we can see why. There is a relatively clean segregation between hot target pixels and non-hot target pixels, even when only considering just two features. The SVM and MLP are able to find this as well and generate an accurate decision boundary cq. distribution of weights. A downside of these methods is that they require some time to generate full resolution segmentation maps. The other machine learning method we implemented, the DNN architecture, fails to perform. We observe very low precision results, due to the network’s inability to generate effective feature maps. These low scores can be explained by the shape of our ground truth data. The fires present in our dataset usually only contain a few pixels. In some cases, there are a few hundred, but spread out and sparse. The architecture we used generates smooth, round, segmentation maps, which we showed earlier as well. While we tried various alternative architectures, with a lower number of features, and convolutional layers, to decrease the daubing effect visible in the segmentation maps, we did not observe any major increase in performance.

An additional limitation was our small dataset. As mentioned previously, convolutional neural networks require large datasets to perform accurately, and even with augmented data our FIRES set was not very sizeable. These shortcomings, in combination with the innate training time of DNNs, make them unfit, in this form, for accurate hot target detection. Other DNN architectures might be able to generate better segmentation maps, although it is hard to rival SVM-based approaches.

References

[1] Running, Steven W. "Is global warming causing more, larger wildfires?." Science 313.5789 (2006): 927-928.

[2] Van der Werf, Guido R., et al. "Global fire emissions and the contribution of deforestation, savanna, forest, agricultural, and peat fires (1997–2009)." Atmospheric chemistry and physics 10.23 (2010): 11707-11735.

[3] Shusterman, Dennis, Jerold Z. Kaplan, and Carla Canabarro. "Immediate health effects of an urban wildfire." Western journal of medicine 158.2 (1993): 133.

[4] Swan, Peter Michael. The Augustan succession: an historical commentary on Cassius Dio's Roman history, Books 55-56 (9 BC-AD 14). No. 47. Oxford University Press on Demand, 2004.

[5] Akyildiz, Ian F., et al. "Wireless sensor networks: a survey." Computer networks 38.4 (2002): 393-422.

[6] Yu, Liyang, Neng Wang, and Xiaoqiao Meng. "Real-time forest fire detection with wireless sensor networks." Proceedings. 2005 International Conference on Wireless Communications, Networking and Mobile Computing, 2005.. Vol. 2. Ieee, 2005.

[7] Giglio, Louis, et al. "An enhanced contextual fire detection algorithm for MODIS." Remote sensing of environment 87.2-3 (2003): 273-282.

[8] Kaufman, Yoram J., et al. "Potential global fire monitoring from EOS‐MODIS." Journal of Geophysical Research: Atmospheres 103.D24 (1998): 32215-32238.

[9] Boschetti, Luigi, et al. "MODIS–Landsat fusion for large area 30 m burned area mapping." Remote Sensing of Environment 161 (2015): 27-42.

[10] Murphy, Sam W., et al. "HOTMAP: Global hot target detection at moderate spatial resolution." Remote Sensing of Environment 177 (2016): 78-88.

[11] Zurita-Milla, Raul, Jan GPW Clevers, and Michael E. Schaepman. "Unmixing-based Landsat TM and MERIS FR data fusion." IEEE Geoscience and Remote Sensing Letters 5.3 (2008): 453-457.

[12] Petropoulos, George P., Charalambos Kontoes, and Iphigenia Keramitsoglou. "Burnt area delineation from a uni-temporal perspective based on Landsat TM imagery classification using Support Vector Machines." International Journal of Applied Earth Observation and Geoinformation 13.1 (2011): 70-80.

[13] Mountrakis, Giorgos, Jungho Im, and Caesar Ogole. "Support vector machines in remote sensing: A review." ISPRS journal of photogrammetry and remote sensing 66.3 (2011): 247-259.

[14] Basu, Saikat, et al. "Deepsat: a learning framework for satellite imagery." Proceedings of the 23rd SIGSPATIAL international conference on advances in geographic information systems. 2015.

[15] Castelluccio, Marco, et al. "Land use classification in remote sensing images by convolutional neural networks." arXiv preprint arXiv:1508.00092 (2015).

[16] Hu, Wei, et al. "Deep convolutional neural networks for hyperspectral image classification." Journal of Sensors 2015 (2015): 1-12.

\n","readingTime":"23 min read","fullURL":"https://pingen.dev/projects/hottarget"},{"slug":"groundcover","title":"Ground Cover Analysis","featured":false,"creationDate":"2016-09-05","category":"Project","excerpt":"Ground cover analysis is used in smart agriculture applications to enable automatic treatment of crops, for instance by finding an optimal distribution of water or pesticide administration. This is especially helpful for farms located in third world countries, that have little access to high-tech machinery and a lot to gain in terms of crop yield. In this project, various methods of ground cover analysis were tested to help farmers in the Bangladesh delta cultivate their land.","labels":[{"id":"research","name":"Research"},{"id":"machinelearning","name":"Machine Learning"},{"id":"geo","name":"Geo"},{"id":"ipcv","name":"Image Processing"}],"cover":"groundcover.jpg","draft":false,"html":"

Introduction

Ground cover (also referred to as FCOVER - Fraction of green vegetation cover), the percentage of ground covered by vegetation in a specified area, is a commonly used metric in agronomic studies. For example to estimate crop yield. Ground cover has been shown to be correlated with biomass yield (measured by weighing dried biomass) by Baret and Guyot. The estimation of ground cover can be used in smart agriculture applications to perform effective automatic treatment of crops, for instance by finding an optimal distribution of water or pesticide administration. This is especially helpful for farms located in third world countries, that have a lot to gain in terms of crop yield.

Farmers in developing regions such as the Bangladesh delta struggle to cultivate their land during the dry season due to soil moisture levels in these times, which are often lacking. Ground water is often salty or briny, which is unfit for raising crop plants. Farmers need to use water pumps to be able to use sweet surface water from rivers and channels, but the pumps are expensive. Therefore, irrigation needs to be optimized.

Effective use of irrigation systems depends largely on the current ground cover and soil water balance. Accurate ground cover analysis could therefore help in developing an advisory system for irrigation scheduling, thereby increasing crop yields.

Ground cover analysis, including estimation (the percentage of an area covered by vegetation), classification (instance-based labelling of vegetation), and segmentation (demarcation and extraction of regions of vegetation), is most often done using image-based methods (though other methods such as thermal analysis also exist). A wide range of computer vision and image processing techniques are applied to monitor and study agricultural changes using both ground level RGB cameras and multi-spectral air- and space-borne radiometers.

Successful ground level image-based ground cover analysis relies on overcoming a number of common obstacles found in agrarian imagery. Lighting conditions can be vary extremely over time. Images obtained during sunny weather can result in different ground cover estimations compared to images of the same field obtained with overcast, or rainy weather. This is due to specular reflection in sunny conditions, or differences in refraction when a canopy is showered in raindrops. Another obstacle is the formation of shadows of the canopy onto itself. Parts of the vegetation that are cloaked in the shadow of leaves or other objects may be classified as soil instead. Especially for large crops that have a high number of leaves, this problem can severely affect classification and segmentation accuracy. Finally, the soil area may be littered with objects that do not fall into either of the two categories, soil or vegetation. Residual waste from other nearby vegetation or animals may influence ground cover analysis, as would small stones, or pieces of wood.

$\"Figure$

Figure 1. Top: Segmentation of rice crops. Bottom: The three types of crops present in our dataset. Mung bean (left); Maize (middle); and Wheat (right). Notice common obstacles for good segmentation: Shade casting, hard lighting, overlapping leaves, and residu on the ground.

An example of crop segmentation is shown in Figure 1 above (top), together with an example of adverse lighting effects and overlapping in our dataset (bottom).

\n\n

Ground cover analysis and crop segmentation has been an active field of research for decades. We will mainly consider the segmentation problem here, because it envelops the classification problem to a degree (if a segmentation map can be generated, pixel-wise classification has either been done already, or trivial to perform), and can be used to perform estimation as well (though segmentation is not a requirement for cover estimation, since we can transform it into a regression problem).

Ground cover segmentation is essentially a two class problem. Pixels in an image belong either to a vegetation class, or to a soil/ground class. Hamuda et al. provide an extensive research on image based vegetation segmentation techniques, broadly categorizing these techniques as colour index based, threshold based, and machine learning based. We will evaluate existing methods keeping to this categorization, but grouping threshold based methods with colour index based methods because the former is mainly performed using colour indices as well.

Colour indices

To a human observer, the most noticeable distinction between soil and vegetation is the colour of these categories. Whilst soil is for the most part brown or grey, vegetation is often mainly green or yellow. This difference in colour is also used in crop segmentation techniques. Most conventional cameras used in ground level field photography will generate RGB images.

One problem with RGB images is that they are dynamic, meaning that the colour of soil varies throughout an image. The soil may be less moist in one place and very moist in another, affecting its RGB values. The same holds for different shades of green on leaves. To give more prominence to the soil/vegetation colour difference this RGB colour space is converted to another colour space, for example by attributing different weightings to the R, B, and G values, resulting in a colour index. In extreme cases, this may directly lead to a binary image that can be used for segmentation. These indices are often generated using either expert knowledge, or machine assisted methods such as fuzzy classifiers.

There exist many proposed alternative colour spaces for estimating various qualities of vegetation. An overview is presented in the following Table.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Index	Formula
NDI	$128 \\cdot ((\\frac{G-R}{G+R})+1)$
ExG	$2g-r-b$ , with $r,g,b$ the percentage of $R$ , $G$ , and $B$ pixels respectively (normalized).
ExR	$1.3R-G$
ExGR	$ExG-ExR$
MExG	$1.26G-0.844R-0.311B$
CIVE	$0.441R-0.811G+0.385B+18.78745$
VEG	$\\frac{G}{R^{\\alpha} \\cdot B^{1-\\alpha}}$ , with $\\alpha=0.667$
COM1	$ExG + CIVE + ExGR + VEG$
COM2	$0.36ExG + 0.47CIVE + 0.17VEG$
CANO	$R/G < P_{1}$ and $B/G < P_{2}$ and $2G-R-B > P_{3}$ , with $P_{1}=P_{2}=0.95$ , and $P_{3}=20$

Table 1. Various colour indices found in literature.

Colour index based methods of vegetation segmentation have the advantage that they are simple, fast and computationally light, but struggle to perform accurately when lighting conditions are bad (i.e. in overcast or sunny weather, or with shadows). This is apparent when one observes the large number of different indices used throughout literature for various purposes. The indices found in the Table above also show such a variation in\nperformance based on the type of images they are applied to.

Usually, these colour indices are used in combination with a set threshold to generate a binary image that can be used for segmentation. The Otsu method is one of the most common approaches to thresholding. It calculates the optimal value to separate two classes by maximizing the inter-class variance (based on a foreground/background histogram). Equation 1 shows the formula of the weighted sum of variances (or intra-class variance) of the two classes that need to be separated. Otsu's method then searches for a threshold that minimizes this intra-class variance (and in doing so maximizes inter-class variance). The class probabilities are calculated from the image histograms.

$\n\\begin{equation}\n\\begin{align}\n \\sigma_{w}^{2}(t) = w_{0}(t)\\cdot \\sigma_{0}^{2}(t) + w_{1}(t)\\cdot \\sigma_{1}^{2}(t) \\text{, with} \\nonumber \\newline \n w_{i}(t) = \\text{Probability of the class separated by threshold \\textit{t}} \\nonumber \\newline\n \\sigma_{i}^{2}(t) = \\text{Variance of the class} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

Other techniques include local dynamic thresholding, hysteresis thresholding, and homogeneity thresholding (deriving a local threshold by calculating local homogeneity from converted greyscale intensity images). Dynamic thresholds increase an algorithms complexity (and therefore influence execution time and memory usage), but often provide better results compared to using only simple colour index based methods with fixed thresholds.

Neural Networks

Machine learning (ML), programming computers to learn from experience, has been applied in a vast range of data-driven research areas for classification, clustering, segmentation, and regression. The model is iteratively corrected, according to the labels corresponding to the input data, through which a function that maps the input data to the desired output data emerges.

More recently, deep learning methods - a branch of artificial neural networks (ANNs) - have gained increasing interest in the machine learning community, and in mainstream media due to the success of AlphaGo (The first AI to ever beat a professional human Go player). ANNs are a type of model inspired by its biological counterpart, hence the name. They are structured as a network of nodes and directed weighted edges, and incorporate an activation function similar to the natural neuron. The input for a given node can consists of one or many connections to other nodes. The input from all input nodes is combined using a transfer function (i.e. summation), and tested against an activation function containing a set threshold. When the threshold is reached, the node will fire, after which its input is available to succeeding nodes. Eventually the network will calculate an output value (or multiple, depending on the number of output nodes), based on its input, weights, transfer functions and activation functions.

Backpropagation is then used to allow the model to learn. When the network has calculated its output value, it can be right or wrong with a certain error margin, which can be defined as an error function. By backwardly propagating this error into the network, it can adjust its weights to provide a more accurate output value in the next iteration. This is done using Gradient Descent, an optimization algorithm that minimizes the error function stepwise by iterating over the training set.

ANNs can be built using many layers, allowing them to learn complex relationships between input data and evaluation data. There are many types of neural networks: Feedforward networks such as autoencoders, and restricted Boltzmann machines (RBMs); Convolutional networks such as AlexNet or R-CNN that have layers that convolve kernels with the image; Recurrent networks such as LSTM (often used in time series analysis); and Recursive networks (recursive autoencoders, for example). Often, a combination of these networks is used, as is the case in convolutional networks that combine fully connected autoencoders, or stack RBMs.

Deep neural networks (DNNs) are a subclass of ANNs that incorporate a multitude of hidden layers, allowing them to learn more complex functions. They are often called Deep Learning methods. DNNs show state-of-the-art results in various domains including natural language processing (NLP), speech recognition, and computer vision tasks like concept-\ndetection and visual question answering (VQA).

The prime component of a convolutional neural network (CNN) is the convolutional layer. This convolutional layer consists of a kernel (smaller than the original input layer) that is convolved with the image, computing dot products. The main advantage of a convolutional layer is that they are able to learn both spatial patterns across multiple input pixels and patterns between pixel channels (R,G,B for example), instead of only the latter.

Often, many of these layers are sequenced together, intermixed with activation functions. Activation functions define the output of a neuron given its input. A simple example of this is the rectified linear unit (ReLu) activation function ( $f(x) = \\max(0, x)$ ), but more complex functions are used as well (SoftPlus, Gaussian). This results in an activation map per kernel.

$\"Figure$

Figure 2. Visualization of features in a convolutional neural network. Activation maps of high scoring layers (right side) are shown for random inputs (left side). Images obtained from: https://arxiv.org/abs/1311.2901.

A feature visualization of convolutional networks is shown in Figure 2. Fully (or dense) connected layers contain neurons that are connected to all input neurons (as is the case with normal neural networks). Pooling layers, such as (soft-)max pooling, can be used a well to downsample and reduce the size of feature representations. Dropout layers, layers that remove nodes based on a certain probability distribution, have also been shown to improve performance because they can prevent overfitting (especially in fully connected networks).

U-Nets

One of the main disadvantages that neural networks have is that training them requires a lot of labelled data, which may not always be readily available. In addition, localization is not the prime strength of neural networks, therefore accurate pixel-wise segmentation is a challenge. To overcome this problem, Ronneberger et al. propose an effective network architecture (U-net) that relies on data augmentation, that allows pixel-wise state-of-the-art classification. U-net consists of a contracting, downsampling pathway that incorporates a number of convolutional layers, and an expanding, upsampling pathway that allows for localization.

Although DNNs are used for semantic segmentation, not much research has\nbeen done on ground cover segmentation using a deep learning approach.

$\"Figure$

Figure 3. U-net architecture. The blue boxes represent multi-channel feature maps. The number of channels is denoted at the top of the boxes, and the shape data is denoted at the bottom. The white boxes in the upsampling pathway correspond to the copied feature maps. Arrows denote various layer operations (see legend). Image taken from Ronneberger et al.

Data

To train and evaluate our machine learning methods for ground cover analysis, we used the extensive labelled dataset produced in the framework of two projects by NWO through an Applied Research Fund (ARF), and the STARS project, a research consortium consisting of the Bangladesh Institute of ICT in Developement (BIID); the International Maize and Wheat Improvement Center (CIMMYT); and the Geo-Information Science and Earth Observation (ITC). The STARS project is funded by the Bill and Melinda Gates Foundation. The NWO projects are funded by the Dutch government.

An 8GB dataset consisting of 2564 images of varying quality and dimensions (around 2000x1500 pixel JPEGs) was hand-annotated by the team in Bangladesh using CAN-EYE imaging software. This dataset is the basis of our ground cover research.

$\"Figure$

Figure 4. Sample data of vegetation RGB imagery taken using a smartphone, and hand-annotated ground truth data using CAN-EYE software.

Using the CAN-EYE software, the Bangladesh team extracted canopy structure characteristics, such a Leaf Area Index (LAI), and Vegetation cover fraction (FCOVER). Various plant species, including Wheat, Maize, and Mung bean, are photographed (see Figure 1 for examples of these crop types, each covering roughly 30% of the dataset). Per plot, around 9 to 10 images are taken in a square formation. Some small overlap is present in these images, but these do not cause any problems for our proposed method since all 9 images are annotated. This is the ground-truth data to which compare our method. From here on out, we will refer to this dataset as CROPS.

$\"Figure$

Figure 5. Frequency of FCOVER value occurence per plant type.

Method

To obtain a score for the colour indices methods, we implemented all algorithms listed in Table 1, namely NDI, ExG, ExR, ExGR, MExG, CIVE, VEG, COM1, COM2, and CANO(PEO), and applied them to our CROPS dataset. Since these algorithms are basic colour space transormations, they require a threshold to do obtain a binary segmentation map. For ExGR the threshold is set at 0, in accordance with literature. For VEG and MExG, we use the mean value of the resulting image as threshold. For the other algorithms, we apply Otsu thresholding. For ExR and CIVE, due to their formulation, the Otsu threshold is taken as an upper bound. For the others, NDI, ExG, COM1, and COM2, this threshold is a lower bound. CANO’s formulation already ensures the generation of a binary map. We leave the P1, P2, and P3 parameters to their default values as reported in the CANOPEO paper.

Applying these methods results in a binary segmentation map of the input image. From these binary images, we can determine the estimation and segmentation errors. The estimation error is determined by calculating the percentage of plant pixels in the binary map, and comparing this to the ground truth FCOVER. The segmentation error can be determined by evaluation of the True Positives, True Negatives, False Positives, and False\nNegatives.

We also use the AP-HI method proposed by Yu et al. for evaluation, due to their high reported performance. After reaching out to the researchers, they were kind enough to share their MATLAB implementation. The pipeline was easily adapted to be able to run on our CROPS dataset. The AP-HI algorithm produces a binary segmentation image, which can be evaluated in the same way as the colour indices methods described in the previous section.

Additionally, we implemented SVM and Random Forest classifiers (just for regression - i.e. calculating the FCOVER percentage), and the previously highlighted U-net neural network (for segmentation). Our network architecture consists of 23 convolutional layers. The final step is a 1 by 1 convolution and Otsu thresholding to produce a binary segmentation map. In our experiments we reduce the number of features to 10 in the first convolutional layer, and increase the amount further down the pipeline (up to a 3x3x80 convolution). We employ Logistic Regression as our loss function.

To deal with the high-resolution data, we split the input into 256x256 pixel tiles, and train/test the network on these samples. The input for the network therefore consists of an 8-bit 3-channel 256x256 pixel RGB image. The output will be a single channel 256x256 pixel binary map. Since we cannot divide the original image into tiles of this shape perfectly, some overlap is apparent. It can be argued that the overlap will introduce some overfitting, but this is overfitting is marginal. The samples are augmented by performing 5 independent semi-random elastic transformations on the original image, and 4 rotations. This boosts the size of our training set, and decreases variance. This results in a total set of 5,101,076 input images, to be split into training and validation data using K-fold cross validation.

We also trained the network using scaled images where no transformations are performed. This has the added benefit of negating any imprecise annotations by downsampling the errors together with the image, while the FCOVER remains the same by definition. By downsampling the original images using a nearest neighbour resampling filter, and applying the same transformation to our ground truth images, we obtain input images of 256x194x3 pixels. We verified that the FCOVER does not change significantly upon downsampling, and found that the mean difference in FCOVER between the original and scaled down version is around 0.0009512. This corresponds to a 0.9512 percentage point difference, which we deemed acceptable for our experiments, since we use the scaled down FCOVER in these cases.

We varied the network's hyperparameters, such as number of layers, and number of features, as part of our experiment. Since the literature on the subject of tweaking deep learning architectures does not provide an optimal strategy for this, we varied the parameters until observing diminishing returns. In similar manner we also evaluated the effect of Dropout at different positions, since this has been shown to increase performance in DNNs.

In addition to a single network for all plant types in our CROPS dataset, we will also evaluated the DNNs performance by training it on each type individually. This resulted in three networks - a Wheat, Maize, and Mung bean segmentation network.

Results

We implemented a number of different methods to be used as benchmark for our machine learning methods. For some of these methods, we conducted additional experiments to gauge the effects of certain parameters on the resulting segmentation or estimation. Figure 6 shows examples of segmentation results for each method.

$\"Figure$

Figure 6. Example segmentations of Maize, Mung bean, and Wheat crops for the given segmentation algorithms (scaled images).

We evaluated the segmentations on Root mean square error (RMSE), and precision and accuracy scores. Overall, Maize seems to be the easiest crop type to segment using these methods. Figure 7 shows the algorithms' precision plotted against their accuracy, split on crop type. The algorithms are able to score highest on Maize images, and lowest on Wheat images. We see that DNN outperforms all conventional methods, although there are instances where another method has a slight edge on the DNN - for example COM2 shows a minor improvement in accuracy for Wheat images. We see that in most cases, RMSE scores correlate with accuracy and precision scores. However, this is not always the case. For instance, we can observe that COM2 has quite a high precision score for Maize images, but the corresponding RMSE is contrastingly relatively high. This discrepancy is due to COM2 underestimating Maize images quite a lot (which, in turn, is caused by the index not being able to recognize very light leaf areas). Since COM2 does not classify many pixels as plant pixels, it does not generate many false positives either.

$\"Figure$

Figure 7. Top: RMSE and standard deviation of scaled images for ground cover estimates for all segmentation methods. Bottom: Precision and accuracy scores for methods of ground cover segmentation of scaled images.

The colour indices methods we implemented provide varying segmentation results. We see the detrimental effects of changing lighting in the leaves, clutter on the ground, and shadows cast by other leaves. We see MExG\nperforming best overall for scaled images. ExG, NDI, VEG, COM1, and COM2 have the biggest trouble with hard lighting changes, as can be seen in the examples above. The other indices, ExR, ExGR, MExG, CIVE, and CANO, overcome this problem. However, this feature also makes them tend to overestimate the crop area, as can be seen in the Wheat examples most prominently due to the density of the crops in this type of vegetation. Their ability to pick up light surfaces has the downside of not being able to distinguish between darker surfaces adequately.

Striking is that the colour index that performs best is surprisingly simple. The MExG index is a weighted linear combination of the R, B, and G planes. We find that MExG only really struggles when dealing with very hard lighting, and near-white surfaces. An example of this can be found in the Maize column, in the area on the large leaf on the left side of the image. We see the error rate increase in images of Wheat especially, since the spikelets are highly reflective. The high blue values in white pixels cause problems for this index. Red values are relatively high even in green leaves, which is why MExG penalizes them strongly. Since colour indices methods do not take into account other information than the R, G, and B values of a single pixel, they cannot infer that white pixels with high RGB values values actually belong to a leaf element in the image, rather than background noise.

The colour indices that perform the worst, most notably ExGR, drastically underestimate the vegetation area, due to their inability to deal with big changes in leaf colour when the light conditions are extreme. On the other hand, ExGR sometimes labels shadows as being part of the plant, resulting in an overestimation of the FCOVER.

The AP-HI method as proposed by Yu et al. performs relatively well for the original images, but not quite as well as, for example, MExG or the DNN. Especially when segmenting images of Wheat plants, it lacks the power to identify the yellowish, lighter spikelets on the top of the plant as being part of the plant. The AP-HI method, like some of the colour indices, struggles to appropriately classify strong highlights, albeit to a much lesser degree. AP-HI performs somewhat worse on our scaled CROPS dataset, as it does on the original set. We believe that due to the decrease in pixel availability in smaller images, the sample size of the hue intensity lookup table decreases as well. In addition, the affinity propagation clustering algorithm has less data to work with, leading to non-optimal results.

For original images, we observed that in many cases the DNN overestimates the plant region, which could be caused by a combination of the input images being rather small, and the large variety in lighting conditions in our CROPS dataset. We did also observe the potential of the DNN to overcome intra-leaf lighting changes. Note that this was with a single network for all 3 plant types.

When we investigate the resulting segmentation map of our scaled images, we see that the convolutional neural network provides segmentations that overcome the problems that hinder conventional colour indices methods. Comparing the DNN to the other methods we clearly see that DNN outperforms conventional colour indices methods, and the AP-HI method. Not only is it the best performing segmentation method overall, we also see top performance for each individual plant type. Especially for Maize and Mung bean we can observe exceptional performance. The DNN obtains similar performance to the well performing colour indices methods for the Wheat type.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Network	All	Maize	Mung bean	Wheat
Single	0.0901 (0.0175)	0.0418 (0.0113)	0.0494 (0.0046)	0.1653 (0.0240)
Typewise	0.0779 (0.0193)	0.0351 (0.0053)	0.0444 (0.0040)	0.1423 (0.0351)

Table 2. MSE results (standard deviation) of a single network versus the scores of the three separately trained networks.

The DNN is able to handle extreme changes in lighting conditions, more so than conventional methods. This is due to the added information of other pixels surrounding the target pixel. The convolutional layer helps in learning these gradual and abrupt hue and intensity differences in a leaf. This does lead to rounded edges in most segmentation maps. In Mung bean images, this is an advantage since the leaves are relatively round themselves. In Maize, but especially in Wheat, this leads to misclassification of a large number of pixels, but the DNN is still able to outperform the other methods.

When looking at the precision and accuracy of the DNN scaled segmentations, we again see that it outperforms all methods. Per individual plant type, there is only one other method that obtains better accuracy: COM2 on Wheat images.

We have trained a array of different network setups. The DNN results shown in Figure 7 show the most optimal performance of our method, which corresponds to three separately trained networks for each plant type.

Estimation

We have explained how segmentation algorithms can also provide an estimation of ground cover. A number of algorithms are able to provide an estimate of the percentage of vegetation in our images without creating a binary map first. They are machine learning methods that are trained on the ground truth cover fraction. Together with the segmentation methods, we show their estimates here. Figure 8 shows the RMSE for each regression method, and the DNN (on scaled images).

$\"Figure$

Figure 8. RMSE and standard deviation of scaled images for ground cover estimates for all regression methods.

All regression methods outperform the DNN. However, the difference in performance is small. In fact, the DNN provides better estimation results for Maize type vegetation, and is on par with the SVM and MLP for estimation of images containing Mung bean plants. The regression methods do significantly outperform the DNN method on Wheat imagery, however.

It is hard to gauge exactly why the regression methods do so much better on Wheat images, since we converted the original image to a percentage based feature vector. We know that our DNN underperforms on Wheat image because it is prone to generate rounded, thick edges, quite the opposite of Wheat leaves. Methods that do direct regression on the FCOVER value do not encounter this problem, which might help to explain the discrepancy in scores.

Machine learning is an effective tool that can be applied to perform more accurate ground cover analysis than conventional methods. The methods proposed in this work outperform existing methods of ground cover analysis, both in RMSE, as well as accuracy and precision. Out of our machine learning methods, the SVM, RF, MLP, and DNN, our DNN architecture shows the best segmentation performance while also being able to provide accurate ground cover estimation.

Limitations

We must note that our neural network approach also has a number of disadvantages. First of all, neural networks, convolutional neural networks in particular, are heavily reliant on properly labelled data. This means that a lot of preliminary work and preprocessing steps are involved with using this type of method for ground cover analysis. In addition, the learning phase is quite slow, especially compared to the colour indices methods that were able to handle high resolution input. Training time was around an hour on an 8-core i7-4700MQ with an NVIDIA Quadro K1100M. This means that for regression tasks, it is probably preferable to use an SVM-based approach, considering the algorithms speed and very accurate estimation results. Second, the architecture we used was not able to handle very large input images. The original image resolution of around 2000x1500px was too large for our network c.q. machine, and we were forced to scale down our input images. Initially, we split our original images into smaller blocks and used those as input images, however, we found that our ground truth data was not always optimal.

We must also address that while we get very good results on networks trained separately on each crop type, this will have implications for the operational use of our methods, and introduce additional costs. A final disadvantage of DNNs is that while they can achieve good performance, it is hard to observe why they perform so well. Methods of visualizing layer activation are currently being researched however to try and overcome this problem.

This brings us to the limitations of our experiments. When we performed visual inspection on our original ground truth, we found that there were many small errors. This was an artefact of doing annotation with the CANEYE software, but difficult to resolve. We expect that the non-optimal ground truth labelling is the reason that the DNN did not perform accurately on the original splitted images. By scaling the original image, and with it the ground truth, we solved many small errors through nearest-neighbour resampling. This leads to a better ground truth, and much better results. We expect that with more accurate ground truth, the DNN is also able to obtain better results on the original (splitted) images. In addition, we expect that with a better machine, larger size input images can also be processed.

References

[1] Baret, Fred, and Gerard Guyot. "Potentials and limits of vegetation indices for LAI and APAR assessment." Remote sensing of environment 35.2-3 (1991): 161-173.

[2] Hamuda, Esmael, Martin Glavin, and Edward Jones. "A survey of image processing techniques for plant extraction and segmentation in the field." Computers and electronics in agriculture 125 (2016): 184-199.

[3] Otsu, Nobuyuki. "A threshold selection method from gray-level histograms." IEEE transactions on systems, man, and cybernetics 9.1 (1979): 62-66.

[4] Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." nature 323.6088 (1986): 533-536.

[5] Cauchy, Augustin. "Méthode générale pour la résolution des systemes d’équations simultanées." Comp. Rend. Sci. Paris 25.1847 (1847): 536-538.

[6] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015.

[7] Yu, Zhenghong, et al. "Automatic image-based detection technology for two critical growth stages of maize: Emergence and three-leaf stage." Agricultural and forest meteorology 174 (2013): 65-84.

\n","readingTime":"28 min read","fullURL":"https://pingen.dev/projects/groundcover"},{"slug":"aves","title":"AVES","featured":false,"creationDate":"2016-01-01","category":"Project","excerpt":"AVES is a video search application created to evaluate relevance feedback in multimedia retrieval systems, for an internship at TNO. The engine uses deep learned neural networks to detect concepts, and is based on the TNO-VIREO, and UTWENTE-PRFUBE systems. It incorporates a novel Rocchio-like relevance feedback algorithm to boost retrieval results, outperforming state of the art reranking algorithms.","labels":[{"id":"research","name":"Research"},{"id":"ipcv","name":"Image Processing"},{"id":"recsys","name":"RecSys"}],"cover":"aves.jpg","draft":false,"html":"

Introduction

Relevance feedback, the application of human feedback regarding the performance of information retrieval systems, has been a topic of study in the field of information retrieval for almost 50 years. Starting with textual document retrieval, the field has since expanded to (content-based) multimedia retrieval (such as audio, images, and video) as well. In the current decade, we are confronted with more and more multimedia content. The high rate at which open sensor networks are being set up and increasing rate at which sensor data is available today expose the need to perform semantic searches over these large heterogeneous datasets.

One of the challenges this brings is bridging the semantic gap, the discrepancy between the high level query posed by the user in natural language, and the low level sensor data features. One way of decreasing this gap is to include the user in the information retrieval system. To identify what the user is looking for, the system can iterate a number of times over its results, using the users feedback on which returned results are relevant for its query and which results are irrelevant. Adapting to the user's feedback, the system can refine the search query and retrieve more relevant results.

In the last years, relevance feedback in information retrieval is an increasingly active field of research. In this section we will give a brief overview of the status quo. One of the most well-known and applied relevance feedback algorithms that has its origins in text-retrieval is the Rocchio algorithm. The classic Rocchio algorithm of relevance feedback looks to find the most optimal query based on the user's initial query, and relevance feedback consisting of positive and negative documents. This method of relevance retrieval found its way from text-retrieval into many other domains, including video retrieval.

Relevance feedback can be performed automatically, without any user interaction, or manually, by allowing a user to mark results as relevant or not relevant in varying degrees. The first branch of relevance feedback, also called pseudo-relevance feedback, has piqued the interest of a large number of researchers and software engineers due to its non-reliance on users, though normal relevance feedback is also very relevant to this day. Whereas in normal relevance feedback the user would have to iterate over the retrieved results a number of times, this does not have to be the case in a (well performing) pseudo-relevance feedback system. This makes it favourable for implementation since often the user does not have the time or motivation to give extensive feedback. Human relevance feedback has been known to provide major improvements in precision for information retrieval system.

$\"Figure$

Figure 1. AVES UI.

Research

In this research, when I was interning at TNO, we evaluated a novel method of multimedia retrieval that incorporates relevance feedback using concept detector weight calibration. The method proposed in this research is based on work by Rocchio. Rocchio has been proven to be a strong method of relevance feedback for text-retrieval and we believe the application for multimedia retrieval purposes is promising and worth investigating. We simulated user relevance feedback (optimal-, pseudo-, and random-relevance feedback) to evaluate the model's performance. We also evaluated the model's performance using human relevance feedback. Retrieval without relevance feedback is taken as a baseline, and the well-performing RS method is used for comparison. Giacinto and Roli introduced this method - that takes into account not only a document's relevance towards the other documents, which is usually considered, but also its irrelevance. The formula they utilize of a document's relevance takes the inverse of a documents distance to the nearest relevant document over the distance to the nearest irrelevant document (see Equation 1 below). This score, named relevance score (RS), is then used as ranking measure.

$\n\\begin{equation}\n\\begin{align}\n RS(I) = (1+\\frac{dR(I)}{dNR(I)})^{-1}\\text{, with} \\nonumber \\newline \n I = \\text{Image} \\nonumber \\newline\n dR = \\text{Dissimilarity from nearest image in R} \\nonumber \\newline\n dNR = \\text{Dissimilarity from nearest image in NR} \\nonumber \\newline\n R = \\text{Set of relevant videos} \\nonumber \\newline\n NR = \\text{Set of nonrelevant videos} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

Data

The dataset that was used contains the full video set from the [2014 TRECVID Multimedia Event Detection track][https://www.nist.gov/itl/iad/mig/med-2014-evaluation] train set. The events in this dataset are all complex activities involving interactions with people and objects. It consists of a number of human activities, including human actions that have temporal and semantic relationships to the overarching activity. The list of events includes, for example: Attempting a board trick; Feeding an animal; Birthday party; Repairing an appliance. The full set adds up to a total of 5594 videos (consisting of, on average, 75.45 keyframes).

Implementation

The implementation for the video search engine has been dubbed AVES (Automatic Video Event Search). It is setup as a Python app using Flask (a lightweight web framework based on Werkzeug and Jinja2). To render the pages it uses Blueprints and a number of small (optimization/minification) tools for static asset preprocessing. This is set up as a Gulp pipeline. Nunjucks is used as templating engine and SocketIO provides a way for the app to communicate changes to and from the backend quickly over websockets. The Figure below provides a schematic overview. We will dive into each element of this overview next.

$\"Figure$

Figure 2. AVES backend.

Initial ranking

The user enters a query, after which the query is ran through a Word2vec model trained on a model of Google News words. Word2vec allows us to create vector representations of words, and thus provides a semantic similarity measure of the query to the various concept detectors.

$\"Figure$

Figure 3. Excellent word2vec illustration by @swatimeena.

The top concept detectors that have a similarity score above a set threshold are then retrieved and used for the following steps.

We trained a total of 2048 concept detectors using deep convoluted neural networks and SVMs. For each video in our test set we extracted 1 keyframe per 2 seconds uniformly from the video. The concept detectors are then used to score each keyframe, resulting in a 2048-dimensional vector for each keyframe (normalized). Background\ndata is also extracted by calculating detector scores on videos in which the concept is not present.

$\n\\begin{equation}\n\\begin{align}\n s_{v} = \\sum_{d \\in D}^{|D|} [w_{d}(s_{v,d}-b_{d})] \\text{, with} \\nonumber \\newline \n v = \\text{Video} \\nonumber \\newline\n d = \\text{Detector} \\nonumber \\newline\n D = \\text{Set of detectors} \\nonumber \\newline\n s_{v,d} = \\max(s_{k_0,d},s_{k_1,d},..s_{k_n,d}) \\nonumber \\newline\n s_{k_{n},d} = \\text{Score for the \\textit{n}th keyframe for detector \\textit{d}} \\nonumber \\newline\n w_{d} = \\text{Word2vec similarity of detector \\textit{d}} \\nonumber \\newline\n b_{d} = \\text{Background score of detector \\textit{d}} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

Videos are ranked according to their score $s_{v}$ (see Equation 2). It is inspired by Aly et al. (similarly using the concept prior and probability measure given confidence) and is derived from the Bayesian model.

In Aly's equation, $E[S(d)|\\vec{o}^{,}]$ is the expected score for a document $d$ given the vector of confidence scores $\\vec{o}^{,}$ , and $w(C_{i})$ is the occurrence probability of a concept given relevance. $P(C_{i})$ is the concept prior, and $P_{\\omega}[C_{i}(d)|\\vec{o}^{,}]$ is the probability measure of a concept occurring in the entire set of documents given the confidence score $o_{i}$ . These parameters correspond to the detector scores, background scores, and Word2vec scores respectively in our equation. We denote $s_{v,d}$ to be the probability of a concept occuring in a video (evidence), and $b_{d}$ be the prior.

In our method listed in Equation 2, $s_{v,d}$ is the score of a given video, for a given detector. It is the maximum of it's keyframe scores (determined by the neural network). This score is then multiplied by the Word2vec similarity score for that detector, $w_{d}$ , and we substract the background score (calculated by running the neural network and determining detector scores on videos that do not contain the concepts) of that detector, $b_{d}$ . This is done for each detector, and the scores are summed. The resulting score, $s_{v}$ is then used to retrieve a ranked list of videos for which the concept detectors that are most relevant to the user's query give the highest score. Video's are returned in descending order based on their score $s_{v}$ .

The concept detectors that were used by the model are also shown to the user in the UI (limited to the best 5 out of a maximum of 30). The next Figure gives a more visual overview of this this process.

$\"Figure$

Figure 4. Overview of the scoring model.

Relevance feedback and reranking

After the videos are retrieved as an ordered list, the user is able to make a selection of which results are relevant for their query. These results are grouped into the relevant set $R$ . The rest of the retrieved results are grouped into the non-relevant set $NR$ . Inspired by methods from Aly, PRFUBE, and Rocchio, we constructed a method of concept detector weight calibration to determine a video's new score: $s'_{v}$ . Equation 3 describes this method of\ndetermining the updated video score.

$\n\\begin{equation}\n\\begin{align}\n s'$

The method uses the following equations. We calculate $w'$ , the recalibrated concept detector weight, by taking the original Word2vec score, $w$ {d} $w d $ , adding the average score of the videos in the relevant set (minus the detector prior, $b_{d}$ ), and subtracting the average score of the videos in the nonrelevant set (again, minus the detector prior). These scores are adjusted with weights $\\alpha$ and $\\beta$ , similar to Rocchio. $\\alpha$ and $\\beta$ were calibrated on sample data ( $\\alpha$ = 1.0; $\\beta$ = 0.5), and also match values commonly used in other information retrieval literature.

This is done for all used detectors, after which the calibrated detector weight, $w'$ , is then plugged back into the initial ranking (Equation 2), where we substitute the original Word2vec score for the adjusted weight. This then results in the video's new score, $s'$ {v} $s v' $ , which is used to retrieve a ranked list of videos with a better precision than the initial ranking.

Supplementary detectors

In addition, we determine the minimum of the maximum detector scores $s_{v,d}$ in $R$ , $s_{minmax}$ , and check for each result in $R$ , if there is a detector score $s_{v,d}$ that is higher than $s_{minmax}$ . This is shown in Equation 4 below.

If there is a detector that scores higher than the minimum of the detector scores in $R$ , we add the detector to our detector set for reranking. The reasoning is that if there is a detector that scores well in the set $R$ , it may be advantageous to use it in retrieval (although it was not picked up with the Word2vec similarity initially).

$\n\\begin{equation}\n\\begin{align}\n s_{minmax} = \\underset{\\forall v \\in R}{\\min} \\underset{\\forall d \\in D}{\\max} s_{v,d} \\text{, with} \\nonumber \\newline \n v = \\text{Video} \\nonumber \\newline\n d = \\text{Detector} \\nonumber \\newline\n D = \\text{Set of detectors} \\nonumber \\newline\n R = \\text{Set of relevant videos} \\nonumber \\newline\n\\end{align}\n\\end{equation}\n$

Method

In order to evaluate the AVES system and answer the research questions as described in the previous sections, an experiment was performed using human participants to provide relevance feedback.

Ten participants with a higher education level (employees and students from TNO, and the University of Twente) were asked to volunteer in the experiment. Participants were excluded from participation if they were dyslectic, had concentration problems, RSI, or were colour blind.

Users used the system for period of up to 50 minutes. The user was asked to search for up to four example queries, serving as an introduction for the user to the system. A list with topics to be queried was then presented, and the user was asked to evaluate the results based on relevance. Users would first perform 16 queries with one method (AVES, or RS), and then switch to do the remaining 16 using the other method. Meanwhile, the user's action and system performance (mean average precision, lookup time, runtime, etc.) was recorded automatically. After the list has been completed, a semi-structured interview was scheduled to evaluate the system on factors such as usability and usefulness (using the well-known System Usability Scale).

Mean Average Precision (MAP) was calculated by taking the mean of the Average Precision (AP) per query. AP for a query is calculated after removing the first 20 results (because in comparing modern information retrieval sysems these are often equal and therefore an uninteresting metric), and the selected relevant results. To signify the difference with normal MAP, we use the following annotation for MAP after the first 20 results and any selected relevant results from the baseline result ranking: MAP∗.

We also evaluated non-human relevance feedback through simulations. Relevance selection is done using optimal- (select N known true-positive), pseudo- (select the first N results as positive), and random- (select N positive at random) selection.

$\"Figure$

Figure 5. Translation of SUS scores into quartiles, acceptability and adjectives.

Results

There was a statistically significant difference between groups as determined by a one-way ANOVA (F(2,27)=18.972, p=.000). A post-hoc Tukey's HSD test was performed to verify intergroup differences, which showed means of all methods differing significantly (p<0.05).

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Method	$\\mu$	$\\sigma$
Baseline	0.1309	0.0102
RS	0.1071	0.0198
AVES	0.1532	0.0155

Table 1. Mean and standard deviation for MAP∗ scores per method.

To get an overview of the precision per event, we calculated precisions averaging over all sessions. A bar plot is shown in Figure 6. We also investigated relevance selection per event. An overview of the True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN) scores are shown in Figure 6. Note that this terminology does not capture the users' beliefs sufficiently, since the user's cannot be wrong in their judgement (if we assume they performed the task honestly). We can state that they are True, or False Positives only relative to the ground truth. A better terminology reflecting the users' honest evaluation might be: Correct positives, Missed negatives, Correct negatives and Missed positives.

$\"Figure$

Figure 6. Top: Average precision (AP∗) per event per method. Bottom: Percentage relevance selection per event per method.

A commonly used measure of robustness is the robustness index (RI). RI is a measure of the ratio of the difference of the number of events that benefit from one method compared to another, to the total number of events. A plot showing the difference in precision between AVES and RS per event can be found in the Figure below. We obtained an RI of 0.5625.

$\"Figure$

Figure 7. Average precision difference (AP∗) between AVES and RS per event.

SUS ratings were $\\mu = 84.00, \\sigma = 10.014$ for RS, and $\\mu = 84.50, \\sigma = 10.462$ for AVES, with no statistically significant difference. Both could be classified as excellent.

Concluding

Precision was found to be higher using the AVES method of relevance feedback than when not using any relevance feedback, or when using a current (well-performing) method, RS. We must note here that when looking at full MAP (including first 20 results) and perfect relevance selection, AVES performs as well as RS.

Lookup time was not found to be lower using AVES than when using RS, but AVES did sport a 75x faster runtime than RS, depending on the distance measure chosen for RS.

References

[1] Zhou, Xiang Sean and Huang, Thomas S, "Relevance feedback in image retrieval: A comprehensive review", Multimedia systems, vol. 8, no. 6, pp. 536-544, Springer, 2003.

[2] Rocchio, Joseph John. "Relevance feedback in information retrieval." (1971). 313-323.

[3] Gia, Giorgio, and Fabio Roli. "Instance-based relevance feedback for image retrieval." Advances in neural information processing systems. 2004.

[4] Aly, Robin. "Modeling representation uncertainty in concept-based multimedia retrieval." SIGIR Forum. Vol. 44. No. 2. 2010.

[5] Brooke, John. "SUS-A quick and dirty usability scale." Usability evaluation in industry 189.194 (1996): 4-7.

\n","readingTime":"15 min read","fullURL":"https://pingen.dev/projects/aves"},{"slug":"dasbot","title":"StarCraft Bot","featured":false,"creationDate":"2015-12-03","category":"Project","excerpt":"Strategy recognition and counter-strategy generation are of vital importance in realtime strategy games. In this research a StarCraft Brood War bot was designed and tested that uses a neural network to learn optimal counter strategies with features extracted from game replays.","labels":[{"id":"research","name":"Research"},{"id":"machinelearning","name":"Machine Learning"}],"cover":"dasbot.jpg","draft":false,"html":"

Introduction

StarCraft: BroodWar (1999) is the expansion set of the game Starcraft (1998), developed by Blizzard Entertainment. It is a Real-Time Strategy (RTS) game, and considered to be, not just in its own genre, a major influence on gaming. The franchise is listed twice (StarCraft, including the expansion, and its successor StarCraft 2, released in 2010) in the top selling PC games of all time, making apparent its relevance even today.

In multiplayer mode, players are required to build a base using harvested resources, build an army, and destroy the opponent player's base with that army. A player is victorious when there are no enemy buildings left in the game. Being an old - but well known strategic game, it lends itself exceptionally well for research purposes. Not only is there a vast number of replays to be analyzed, it also runs smoothly on any modern computer setup. This allows for automated tournaments to be held online versus different automated agents.

The focus of this project was to implement such an automated agent using a model for strategy prediction that relied on data mined from replays. The main heuristic used to select counter strategies was a time-optimal counter score calculated by a number of different offensive and defensive features.

Competitive StarCraft

Although the best StarCraft game AI's cannot yet outperform expert human players, the competition grows stronger every year. One of the main stages for those game AI's (or bots) are automated tournaments. Currently, the three largest tournaments of this kind are the Artifcial Intelligence and Interactive Digital Entertainment (AIIDE), the IEEE Computer Intelligence and Games (CIG), and the Student Starcraft AI (SSCAI) competitions. In these tournaments competetors can submit their bots, after which a set number of games will be played,pairing each bot against eachother via LAN.

Research

Weber et al. (2009) have used data mined from replays to predict strategies and develop an opponent model. Their results show that machine learning algorithms can be used to successfully predict strategies with 69% precision 5 minutes into the game, growing to 86% at 10 minutes (using NNge, or non-nested generalized exemplars). Using a rule set to classify tier 2 (or mid-game) strategies, they labeled their replay data based on domain knowledge.

Synnaeve & Bessiere (2011) have also made progress in the area of Bayesian networks\napplied to build order prediction, and reactive build order construction. In one study they computed the distribution of build orders and applied a Bayesian network to generate an estimate of a players build order based on the observed game state. Prediction results were shown to be as high as 4 buildings ahead of a current game state, dropping to 3 when a 0.3 random noise ratio was applied to simulate a competitive setup.

Bot Overview

The bot performed in a highly modular fashion, as is common in these types of bots (although a holistic approach has been attempted in RTS games - however none as complex as StarCraft). This modularity allows it to break down the problem of winning a game into seperate subproblems and deal with them individually. To achieve this, a number of managers were implemented, each handling a different aspect of the game.

$\"Figure$

Figure 1. A modular overview of the managers in the bot. Solid arrows indicate control, whereas dashed lines indicate data flow. Managers within a column also have data flow connections though they are not shown here.

The most important managers are the Build Manager; the Macro Manager; the Micro Manager; the Scout Manager; and the Strategy Manager.

Features

Various features were implemented for each of these managers, such as an extensive build- and economy planner for the build/macro manager (responsible for base and worker construction, and resource harvesting); a squad system to optimize army movement and unit positioning, and a potential field implementation of target acquisition for the micro manager; a system to indicate optimal time and location for the scout manager; and finally the neural network based model for counter strategy selection for the strategy manager.

$\"Figure$

Figure 2. Overview of the three different potential field implementations. The Global Field is the top layer and represents the difference between walkable and nonwalkable tiles, shown here in green and black respectively. The Squad Field shows a group of Dragoons attacking some Drones, wanting to move in attack range (green), and getting repelled a bit by eachother (red area). The bottom image shows a set of Personal Fields, two of which have been reversed as a consequence of the unit having their cooldown activated.

Data

A set of 24552 Protoss versus Protoss (or PvP) replays were gathered from the website bwreplays, which was reduced to a number of 24485 (99.727% of the original dataset) after pruning some faulty or otherwise unusable replays. To enable modeling of the various strategies (or build orders) that exist in the game, only PvP matchups were considered. The reason for\nthis is twofold. Firstly, the bot was created to play the Protoss race. Hence replays including Protoss players. Secondly, choosing the option to consider mirror-matchups only means a separate Strategy Tree for the opponent race does not have to be created. It is expected that results from this project can be applied to different races as well; taken into account that separate Strategy Trees are produced from those replays.

The dataset contains replays by 11984 unique players, both from individual games and tournament settings. Spanning from 1999 to 2013, and versions 1.08 to 1.16.1 (current) of the game, it covers the game's entire lifetime. This enabled modeling of older, though maybe not used as often, strategies as well.

Together with the creation of the Strategy Tree, a database was created to log all the data regarding offensive and defensive power for each player according to the feature set and the id of the winning player, per replay. These values were also stored in the Strategy Tree, but they were stored in a separate database because they are used separately in training the neural network.

Since only offensive and defensive values were used for the current meta-game (they may have been altered in patching throughout the years) to determine a counter-strategy, only replays from the current patch (1.16.1) were taken into account. This means that the training set contained a total of 10889 entries (44.472% of the original pruned dataset).

Strategy Tree

The Strategy Tree is a hierarchical tree ADT on which nodes represent a distinct choice in build order. This could be the construction of a Forge (a Protoss building that unlocks the construction of static defences), or the researching of an upgrade. The race's main building will be represented as the root node, since this is the building every player starts off with and it is always present when the game begins.

From there on out it branches into the different decisions in build order a player has made. Each path in the tree then represents a strategy for that race, and the entire tree is a model for all the strategies observed in the replays.

For every replay parsed, basic information on the player names and id's, the winning player, and offensive and defensive power for both players is stored. The replay is then fed into a tree generating module, which gathers data on the build orders from both players. It will traverse through the existing tree,\nstarting at the root node, and go down a path if the actions of the build order from the replay match the ones found in the tree. If it cannot find an existing path, it will create a new branch. The likelihood of a specific branch is also updated. This value represents the average ratio of a node being visited compared to its siblings.

In theory, the Strategy Tree could grow infinitely large. However, since the tree is generated based on real-life replays this is not the case. All paths in the tree represent a build order executed in a live match, or a subset of one such replay. There are a number of game decisions not included in the tree as nodes, such as Pylons, Assimilators, and Photon Cannons.

The final element of the Strategy Tree is the set of values for the offensive and defensive power stored in a node. These values represent the ability of the strategy to combat its opponent, either by striking at them or by defending from an attack. Offensive power is dependent on the combined raw attack power of the units present, the number of attack upgrades researched, and the number of unit-producing structures. Defensive power is dependent on the combined raw armour power of the units present, the number of defence upgrades researched, the number of static defences, and the number of worker units (as an indication of resource income).

$\"Figure$

Figure 3. Building the Strategy Tree. A build order gathered from the parsed replay is added to the existing tree by traversing its nodes until no existing path can be found. The existing branch is then continued until the entire build order is represented in the Strategy Tree.

Strategy Recognition

Once the Strategy Tree has been generated, it can be serialized and used by the bot in the Strategy Manager. Whenever the Scout Manager gains new information on the status of the opponent, the Strategy Tree is used to make an assessment on what strategy the opponent is pursuing. A list of currently observed opponent units and upgrades is fed into the Strategy Tree model, and returned is a set of paths that contain the observed build order. For example, when a Nexus and Gateway are observed, a list of paths containing this subset is returned. From that list the smallest option is selected, meaning that a path of lesser length still containing the observed subset is preferred over options of bigger length. An observation can match two separately returned paths at the same time. The recognition of an enemy's position in the tree is obviously dependent on the validity of scouted information.

Strategy Generation

A neural network trained on the value of the winning player was applied with a supervised learning paradigm using the Encog framework, and tested with 5-fold cross validation (to allow it to train in the same environment as the\nbot and strategy model). The network consists of 15 input neurons (7 offensive/defensive features for each player -as listed above-, and the total iteration count), 2 hidden neurons, two output neurons, and two bias neurons. The bias neurons are set to random values in the range of [0.3, 0.7].

The neural network was able to predict a winner of a match given these input features with a validation error stabilizing at a mean of 0.2115 over the last 2000 iterations, meaning classification of about 78% of the replays to their correct output (win or loss condition).

The weights learned by the network were then applied to the bot and utilized in its Strategy selection model.

$\"Figure$

Figure 4. Error plotted against the number of epochs of the 5-fold cross-validation testing, stabilising at a mean of 0.2115 over the last 2000 epochs. The light blue plot is the original plot per epoch; The dark blue plot is a plot averaging every 10 epochs; The red plot is a plot averaging every 100 epochs.

References

[1] Weber, B. G. and Mateas, M. (2009) A data mining approach to strategy prediction. Lanzi, pp. 140-147.

[2] Synnaeve, G. and Bessiere, P. (2011) A bayesian model for opening prediction in rts games with application to starcraft. Cho et al., pp. 281-288.

\n","readingTime":"11 min read","fullURL":"https://pingen.dev/projects/dasbot"},{"slug":"guiderobot","title":"Guide Robot","featured":false,"creationDate":"2015-11-21","category":"Project","excerpt":"This project was aimed at researching the effects of head rotation and audiovisual feedback in guide robot behaviour. A robot was constructed for the purpose of guiding small groups of people around in an unfamiliar area. This research was conducted as part of the EU-funded SPENCER project (600877).","labels":[{"id":"research","name":"Research"},{"id":"robotics","name":"Robotics"},{"id":"ux","name":"UX"}],"cover":"guiderobot.jpg","draft":false,"html":"

Introduction

Handling transferring passengers efficiently poses a significant challenge in the commercial aerospace business. In 2013, the number of transfer passengers at Amsterdam Airport Schiphol - the largest airport in the Netherlands - rose to 22 million, totalling more than 40% of all passengers. It is clear that managing this massive migration of people effectively is of benefit for both the passengers as well as airliners. Passengers may get lost, have trouble finding their gate, or run into language barriers. This is just as small set of reasons for flights to be seriously delayed, and passengers missing their connecting flights.

A guide robot could be of help to passengers in need of assistance with finding their way around the airport, and it is exactly this purpose for which the SPENCER project was instigated. This EU funded project is doing research on the implementation of a robot platform that will be able to guide groups of transferring passengers at airport hubs, and by doing so decrease the current transfer time.

Research

As part of this project it was the aim of this study to research the optimal behaviour of a guide robot, most notably in a crowded space such as an airport terminal. The interest of the study was to research the effect of head rotation and audiovisual feedback in guide robot behaviour.

In agreement with results from Butler & Agah (2001), Joosse et al. (2014) found in their research on Human-robot interaction performed with Chinese, American (US) and Argentine participants that, across cultures, users prefer a robot not to enter their personal space. Partially in accordance with previous studies on cultural differences in human interaction it was found that in the high-contact condition of Chinese participants, a closer approach was perceived as more comfortable. Argentinian participants however, though being a high-contact culture as well, responded more similarly to participants from the US, which favoured a less close approach. This could be due to Argentina being a relatively individualistic country compared to other South American countries. Participants found it more appropriate when the robot positioned itself between the mother and child in a triangular setup of mother-father-child with the robot approaching the group, than between the father and child. This could be attributed to the robot facing the father (alpha actor). This was taken into account in designing the robot's locomotor behaviour.

Shiomi et al. (2010) performed a guide robot experiment in the field in an attempt to attract more bystanders into overhearing its conversation, and thereby spontaneously participating in the tour. In the experiment, a guide robot was set up to greet, guide, and give advertising information to groups of people in a shopping mall. Results indicated that participants listened more when the robot was travelling backwards and\nthat it attracted more bystanders to listen in, though the ratio of people being guided might be higher in the forward condition. This effect may be explained due to the area of audience being in front of the robot in\nthe forward condition, meaning bystanders that are opposite to the robots movement direction have a hard time entering the area of audience. In contrast, when the robot is driving backwards, the area of audience is turned towards the group. This means that bystanders coming from behind the guiding robot can more easily enter the area of audience. Since airport guide robots should exhibit the same kind of behaviour as the tour robot in this study it was implemented, though care was taken to ensure that the rate of people arriving at the intended destination remains high enough, since attracting people should not be a top priority.

Robot platform

The mobile robot system used was a Festo Didactic Robotino v2.0. On top of the base system an MDF platform was constructed to attach the outer shell, putting the robot's total height at 170cm. This shell, which was made up of a metal frame and a combination of plastic and synthetic material, held a tablet which was used to display output to, and receive input from, the user.

Hardware

The platform's hardware components included a stainless steel chassis where the power supply, the drive module (3 independent omnidirectional drive units mounted at a 120 degree angle to one another) and the distance and bumper sensors were located. On top of that a command bridge where the controller and I/O module were located;

$\"Figure$

Figure 1. Robot plaform interior diagram.

Software

The code running on the controller unit reads odometry data from the motors and Com (Command Bridge), and measures the distance travelled. The motors output the x/y coordinates with respect to origin. This data is sent through a socket to a TCP client running on the tablet, where the remaining distance is displayed after some post-calculations. Voice generation and recognition is done using the Google Android Speech library, relying on keyword recognition.

Experiment

A between-subject study was performed with 25 participants to test the effects of robot head direction and rotation. The robot would guide groups of up to 3 people around an unfamiliar area at the University of Twente in different head rotation conditions accompanied by brief verbal dialog. Another iteration of the experiment (N=19) tested the optimal modality of audio-visual feedback during the guiding process.

Questionnaire results together with comment and video analysis revealed that participants evaluated the robot facing them and driving backwards as unnatural and intimidating. If information relating to the guide process is shown, backwards locomotion is rated more favourably. Furthermore, participants much preferred touch input to voice as a desirable way of interaction with the robot.

$\"Figure$

Figure 2. Participants engaging with the robot.

References

[1] Schiphol. (2014) Transport and traffic statistics. Schiphol.

[2] Spencer. (2011) European research project: Cognitive systems and robotics. Spencer.

[3] H. Hicheur, S. Vieilledent, and A. Berthoz, "Head motion in humans alternating between straight and curved walking path: combination of stabilizing and anticipatory orienting mechanisms", Neuroscience letters, vol. 383, no. 1, pp. 87-92, 2005.

[4] J. T. Butler and A. Agah, "Psychological effects of behavior patterns of a mobile personal robot", Autonomous Robots, vol. 10, no. 2, pp. 185-202, 2001.

[5] M. P. Joosse, R. W. Poppe, M. Lohse, and V. Evers, "Cultural differences in how an engagement-seeking robot should approach a group of people", in Proceedings of the 5th ACM international conference on Collaboration across boundaries: culture, distance & technology. ACM, 2014, pp. 121-130.

[6] M. Lohse, N. van Berkel, E. M. van Dijk, M. P. Joosse, D. E. Karreman, and V. Evers, "The innfuence of approach speed and functional noise on users' perception of a robot", in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE, 2013, pp. 1670-1675.

[7] M. Shiomi, T. Kanda, H. Ishiguro, and N. Hagita, "A larger audience, please!: encouraging people to listen to a guide robot", in Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction. IEEE Press, 2010, pp. 31-38.

[8] K. M. Lee, W. Peng, S.-A. Jin, and C. Yan, "Can robots manifest personality?: An empirical test of personality recognition, social responses, and social presence in human-robot interaction", Journal of communication, vol. 56, no. 4, pp. 754-772, 2006.

\n","readingTime":"7 min read","fullURL":"https://pingen.dev/projects/guiderobot"},{"slug":"halo","title":"Monitoring Panic Attacks","featured":false,"creationDate":"2015-09-13","category":"Project","excerpt":"Panic disorder (PD) can be effectively treated with a variety of interventions, including psychological therapies and medication. In the great majority of cases hyperventilation is involved, exacerbating the effects of the panic attack. Deliberate deep breathing exercises help to rebalance the oxygen and CO2 levels in the blood. The aim of this project was to create a smartwatch app that monitors and aids patients suffering from PD.","labels":[{"id":"research","name":"Research"},{"id":"mobile","name":"Mobile"},{"id":"health","name":"Health"}],"cover":"halo.jpg","draft":false,"html":"

Introduction

The fifth (and latest) edition of the APA’s Diagnostic and statistical manual of mental disorders (DSM-5) - globally used as a manual for diagnosis of mental disorders - describe anxiety disorders as "...disorders that share features of excessive fear and anxiety and related behavioural disturbances.". Broadly distinguishing the two, whereas fear is a response to a perceived immediate threat, anxiety is a response to a future threat. Physical and mental symptoms include surges of arousal relating to the fight or flight response (fear), and muscle tension (anxiety).

Panic attacks are a specific type of anxiety disorder and manifest as a sudden surge of fear or distress. These attacks usually build up and reach a peak within minutes after onset, during which symptoms such as accelerated heart rate; sweating; trembling; feelings of choking; feeling dizzy; nausea; chills or heat sensations; may be experienced. Patients often report a fear of losing control or ’going crazy’, and fears of dying. About 2 to 3 percent of US and EU adults and adolescents are estimated to suffer from panic disorder (PD), afflicting females more than males (at a ratio of approximately 2 to 1). The median age at which patients describe symptoms of panic disorder is 20 to 24 years.

It is obvious that patients suffering from PD experience extreme discomfort in their daily lives. Not only is the attack itself very stressful; the worry about future panic attacks and (often unfounded) worry that a more severe disease is the cause of the attack (such as cardiac disease or cancers), can hamper a patient in their everyday routine. Embarrassment for the physical and mental symptoms may also lead to avoidance behaviour, restricting social contact. Patients may not want to (re-)visit areas that trigger panic attacks. These areas are usually crowded and hard to get out of quickly such as supermarkets or bus and train transportation. More alarmingly, absence from medical facilities and work floor seriously deteriorate the patient’s quality of life.

Since stress is linked to increased heart rate, the application proposed in this research could also be a positive contribution to the quality of life for people dealing with stress. Stress is present in everyone’s life, be it from professional life or personal worries. A little bit of stress is a commonality, but prolonged and intense stress may cause feelings of fear, or can lead to a burnout. Therefore it is important for people to keep an eye on how stressed they are, and relax when possible if they feel too stressed. Mindfulness is a kind of practice that could be of great benefit to reducing stress levels. This form of mediation teaches you to let your thoughts come and go again, whether they are positive or negative. Stress can lead to people over-analysing their troubles, and getting sucked from one anxious thought into the other by association. Mindfulness therapy aims to teach people to live in the here and now, without worrying about failure (now, or in the future). This allows for better understanding of the negative thoughts, and creates time to reflect on the severity of the perceived problems.

Treatment of patients that experience anxiety or mental stress varies from breathing exercises (although these are used less frequently today) to physical exercise to iterating through a list of small tasks. Breathing exercises are aimed at restoring normal breathing patterns. Physical exercise training is focussed on developing the sense of recognition in patients. Since symptoms of anxiety attacks and stress are very similar to side-effects of physical exercise (such as increased heart rate, sweating, etc.), this type of treatment enables patients to prepare for, and become more resilient to, the sudden surge of discomfort. Another type of treatment focuses on distracting the patient from their bodily functions. The experience of an anxiety attack turns a patient inwards, attentive of their body. Distraction exercises provide the patient with simple tasks such as ”What is the song called that is currently on the radio?”, or ”What does it smell like in here?” to shift their attention away from their body.

It is important for the patient to recognize that anxiety attacks themselves are harmless. Research done in the field of panic and anxiety disorders link low heart rate variability to panic disorder. Heart rate variability (HRV) is defined as the variation in the time interval between consecutive heartbeats and can be detected using for example ECG, or blood pressure analysis. Patients with anxiety disorders have been shown to have a reduced HRV, meaning a more stable pattern of intervals between heartbeats.

Towards a digital assistant

A panic attack itself is characterized by increased heart rate and sweating. These are factors also prevalent in increased mental stress, which may be measured and analysed by a smartwatch. Not only is wearable technology such as the smartwatch a tool that is easy to use and quick to access, it also provides a non-intrusive method to detect panic attacks.

Patients diagnosed with anxiety disorders and more specifically panic disorder can currently receive treatment in the form of therapy sessions with trained psychologists. As part of these treatments, patients are often challenged to do exercises at home as well. For patients with agoraphobia (fear of open and busy spaces) this may consist of visiting a local shopping mall, or taking a busy metroline. Patients that experience panic attacks can be given a short list of tasks that help them distract themselves from the panic attack when it strikes, or homework exercises to help their respiratory pattern.

Wearable technology can possibly help these patients in detecting, helping them during, and recovering from, an anxiety attack or stress. Wearable technology is still up and coming and it is clear that not much research has been done in this field. Through a small feasibility study one learns that there are already apps out there on the consumer market that track a user’s heart rate, as well as HRV. However, none of these focus on coping with anxiety or panic disorders. There are some apps that try to decrease stress, but none of these utilize the smartwatch capabilities and the way of presenting therapy exercises proposed in this research.

\n\n

As described above, not much academic research has been done specifically for these types of applications. However, a number of commercial applications have been developed. We will discuss a number of them now to give an overview of the relevant related applications in the current field.

Salahuddin et al. investigated the possibility of using ultra short term analysis of heart rate variability in mobile applications using ECG. 24 subjects were tested during a baseline reading and an experiment reading consisting of the Stroop color-word test. Findings indicate that mental stress levels could be accurately measured within 50 seconds.

Work by Muaremi et al. presents a solution for stress monitoring using a smartphone that uses features from both the phone as well as wearable belts. Audio, physical activity, and communication data were recorded during the day, and heart rate data was collected at night. 35 subjects were tested over a period of 4 months to evaluate the system. This yielded an accuracy rate of 61% for all features combined for stress level classification (3 levels: low, moderate, and high perceived stress). Using just the features from the smartphone, the accuracy dropped down to 55%.

In the commercial world, examples of similar applications include Samsung’s S Health app, and Embrace, a project that recently went live on crowdfunding website Indiegogo. Due to the novelty of these products it is hard to gauge their effectiveness, but we will nonetheless discuss these examples. S Health uses an oximeter to measure heart rate data, and calculates HRV to determine stress levels. Embrace is an upcoming monitoring system that combines a wearable watch with a smartwatch and web application that measures mainly electrodermal activity (autonomic data that is activated by regions deep in the brain involved in emotions such as fear, anxiety, and positive excitement).

Digital assistant

The system that was developed for this research was dubbed Halo (after the Latin verb for to breathe). It consists of an Android mobile application which serves as a platform on which to send and view therapy exercises; an Android wearable application which serves to monitor the user’s physiological data and receive notifications from the mobile application; an online web application that aggregates the collected data and generates a visualization of that data; and a backend to and from which the various applications may read and write data. Therapy exercises that are presented on the mobile and wearable applications during moments of heavy anxiety or stress were designed in collaboration with experts in the field. To test the implementation and study effectiveness, participants were asked to use the system for 2 full days, after which a post-experiment interview was held to evaluate the system. We will discuss each of these aspects of our methodology to a greater extent of detail in the subsections below, starting with the therapy exercises.

Therapy exercises

Together with two experts in the field of Psychology, Health, and Technology at the University of Twente a number of therapy exercises were developed that could reduce stress levels for users of the Halo system. These exercises were generated as an Android notification on the mobile application, and in a shorter form served on the wearable application as well. An example of the exercises used for this experiment was the following.

Notification as served on the wearable application:
\n
Listen to how many different sounds can you hear right now.
\n
Notification as served on the mobile application:
\n
Relax, and try to listen to how many different sounds can you hear right now. First take in the most prominent sounds, the ones with the loudest volume. Discover what other sounds are hiding behind that. Try closing your eyes to focus even more on what you are hearing.
\n

The aim of the exercises is to decrease the users’ stress levels and thereby the heart rate of the user. The exercises find their roots in mindfulness therapy, which has been elaborated upon in the previous section. A set of 15 such exercises were designed for the purposes of this experiment, each containing both a short as well as a full-length version. Part of the exercises contain breathing exercise elements, and most ask the user to be (briefly) more aware of their surroundings. Closing of the eyes is also encouraged if possible, to enhance the stimulation of the other senses.

Exercises more directly connected to the user’s heart rate were considered, such as showing the current heart rate on a radial plot, and also showing the optimal heart rate on that same plot. This would allow a user to discover optimal behaviour (such as breathing patterns) that leads to reduced stress and/or anxiety levels. These exercises were not implemented in the final iteration of the system however, after receiving expert feedback stating that this would likely lead to a user concentrating even more on their own bodily functions (and thus leading to increased stress/anxiety). Instead, the final iteration of the exercises’ design focusses on turning people outward and towards the world, providing a safe vantage point from which to reflect on what is causing one’s stress.

Implementation

In this section we will expand on the implementation of the different components of the Halo system. We will highlight features of the wearable application (based on an open source heart rate monitoring algorithm), the mobile application, and the web application, respectively.

The mobile and web application are tied to a backend using the Parse cloud platform that contains a database, a registration system, and a login system. A graph showing these connections and dependencies is shown in Figure 1.

The wearable application saves sensor event data into the shared data layer, and is able to notify the mobile application of changes in the data layer, or open a notification, through intents transmitted over Bluetooth. The mobile application, in turn, pushes notifications to the wearable device over Bluetooth. It also stores pulse data, metadata, and other information registered during the experiment to an external database. The web application retrieves this data to build a visualization model. It also features a registration system that connects to the database.

$\"Figure$

Figure 1. Halo system overview.

Wearable application

The wearable Android application is adapted from the Moto360 monitor and utilizes, besides the heart rate sensor to monitor a user’s heart rate, the (significant) motion sensors, step detector sensors, as well as humidity sensors to detect events related to sporting activity or other types of activity that could increase a user’s heart rate but are not stress-related.

During these periods of time, sensor data indicating a high stress level moment is disregarded. Sensor events are fired a couple of times per second and registered on the smartwatch application. Sensor events have raw sensor data associated with them (i.e. in the case of a heart rate sensor event, the beats per minute), as well as an accuracy value, and a timestamp for the event. The accuracy value is a value between (and including) 0 and 3, and indicates how accurate the reading was.

To ensure that only reasonably accurate data is used, all sensor events of accuracy 0 are disregarded. Sensor events of accuracy 1 were opted to be used for this experiment, due to the occurrence of periods of sparsity in sensor events with a high-level accuracy (> 1). Sensor event data is aggregated every 10 seconds, averaged, and stored as a DataItem into the shared storage existing between the wearable and mobile Android applications. A DataItem is synchronized across all devices in an Android Wear network. The mobile application is then notified of a change in this storage, and the new values can be retrieved.

The interface of the wearable application shows only the Halo logo, since for the purposes of this experiment, the watch is used merely to record physiological data, and show notifications. Once a stress event is fired (more on this in the following subsection), the notification will be pushed to the smartwatch. The user is then able to read the (short form) therapy exercise, and can opt to open it on his/her smartwatch as well to read the long form. A visualization of this is given in Figure 2.

$\"Figure$

Figure 2. Left: A therapy exercise is visible on the smartwatch application. Right: An action button allows a user to read the longer version of the exercise on their mobile application.

Mobile application

The mobile component of the Halo system consists of the Android application. This component reads the DataItem’s pushed from the wearable application into the shared storage after being notified by the wearable application. From this, it retrieves periodic sensor data, such as heart rate data (in beats per minute), and step detector data (which triggers an event each time the user takes a step). Physiological is retrieved from storage about every 10 seconds, depending on the accuracy of the readings.

The interface of the mobile application is designed to be very straightforward and user friendly. One switch starts and ends a baseline reading, and another starts and ends an experiment reading. A purposeful decision was made not to show feedback to the user regarding the physiological readings, since this may in fact trigger an anxiety attack or raise stress levels further. A visualization of the mobile UI is shown in Figure 3.

Sensor data related to a user’s pulse (heart rate), is stored from the mobile application using the Parse backend into a database. Besides heart rate, a timestamp and reading type (baseline or experiment) are also stored. Metadata such as the average heart rate, and standard deviation is also saved to the database. This information is used to determine when a user should receive a notification with a therapy exercise. Once a user’s heart rate exceeds the threshold of 2 standard deviations above their mean heart rate and a gradient is detected in heart rate values (i.e. there is a significant rise in heart rate over a short period of time), a stress event is registered, which triggers the generation of a therapy exercise notification. Statistically this will happen in about 2.5% of pulse measurements. Given an average of 53.54 pulse measurements in 8 hours, this would result in 1 to 2 peak moments on average. A therapy exercise is then selected at random from a list of 15 exercises in total. Once a notification has been triggered, no subsequent notifications were generated within the next 30 minutes. This refractory period was implemented to prevent the notification system from firing too often when the sensors return inaccurate data. Another reason is that the therapy exercises often take a minute or two to complete, and that a user would probably not have time to continually preform these exercises if they follow each other too rapidly.

Once the notification has been pushed to the smartwatch, the user is able to open it on their mobile phone as well. If this action button is pressed, the application will fire up and display an extended version of the therapy exercise. Since the mobile application provides more space to work with, and room for a user to navigate, additions such as a favourite exercise system can be implemented more easily in contrast to the wearable application.

$\"Figure$

Figure 3. The interface for the mobile application. Left: Baseline reading button highlighted. Right: Experiment reading button highlighted.

Web application

A user may view his or her recorded physiological data through the Halo web application. The web application consists of a landing page on which the various components of the system are showcased, a registration/user system that allows a user to sign up and use the web application securely, and a dashboard page. A user manual that explains how to operate the devices and use the system, as well as terms and conditions for using the system may also be found here. The dashboard page plots the user’s physiological data. The vertical axis shows the BPM values of each measured Pulse object, and the horizontal axis shows the timestamp at which this value was received by the mobile application and sent to the external database. Figure 4 shows a screenshot of the dashboard page.

Since a user’s mean heart rate is saved during the baseline reading, it is used in the visualization to indicate areas of the curve with BPM values that exceed this value (marked orange), and areas with BPM values that fall below this line (marked blue). By default, the view is set to that of half a day, but the entire history can easily be viewed, and one can zoom to a specific point in time to inspect that part of the graph with greater care. Also shown in Figure 4 is the button to turn on live updates. Switched on, the graph will only show the latest 5 pulse measurements. This could be useful for real-time therapy sessions, where it is less important to view the activity of a long period of time, but rather the immediate response in stress is considered. The web application was shown to the participants after the second day of the user evaluation and before the post-experiment interview. This was done to simulate such a therapy session setup, and to prevent the observance of the recorded data interfering with the user’s self-reported stress levels.

The web application was constructed to serve as a guide to both therapists and patients dealing with anxiety and stress, as well as normal people striving to decrease their stress levels. Since all physiological data is timestamped, it is easy for a user to review his or her day. While doing so, he or she could recap critical moments at which a rapid increase in heart rate was measured. Reflecting on this, perhaps in the company of a therapist, one could then focus on why this occurred at this exact moment. This is crucial because patients often have trouble distinguishing what event lead to the anxiety attack or stress, as described in the previous section. The dashboard might help with exposing these underlying factors.

$\"Figure$

Figure 4. Fragment of the dashboard page of the web application. Shown is a graph of a user’s heart rate (BPM versus time), as well as an option to turn on live updates (showing only the latest 5 pulse measurements). Values in the plot above the measured mean heart rate are coloured orange, and values below the mean heart rate are coloured blue.

User evaluation

An experiment was set up using the implementation of the Halo system as described in the previous section. As noted before there is difficulty in testing an application such as this in a clinical setting with patients suffering from anxiety or panic disorder in the scope of this research project. For this reason, experts in the field (rather than patients) were asked to participate, and a subjective post-experiment interview with accompanying questionnaire was used.

Participants were invited to participate in an experiment lasting two days. On the first day, a baseline reading was performed, and on the second day an experiment reading was performed. During the baseline reading, a series of pulse readings were tracked for an average period of 8 hours over all participants. The participant was instructed to keep wearing the smartwatch for as long as it was comfortable, and that if he or she might find it wearisome or annoying to wear, the reading could be stopped prematurely as well.

During the experiment reading, similar pulse readings were tracked. The participant was instructed to wear the smartwatch for the same amount of time as had been done during the baseline reading. When an anxiety or stress event had been detected, a therapy exercise would be provided, as previously described in the previous section. In addition to these exercises, the participants would also receive therapy exercises at 5 random points during the reading (relatively evenly spaced out) as a control measure for evaluation and to ensure that at least some messages were generated in case participants would remain very relaxed.

The participant was instructed to follow these instructions as he or she would see fit. The participants had been given an activity form to be filled out after the end of the experiment reading. On this form, the participants were asked to draw a graph of their perceived heart rate on the second day ofthe experiment, and indicate moments of activity or events and the kind of activity or events. The graph was to be drawn on a timescale from waking up to going to bed. The marked events were also to be placed on a circle scale to indicate how stressful this event was for the participant. After this, the participant was invited for an interview.

Results

Participants evaluated the application as High Excellent (89.06 System Usability Scale (SUS) score). These were the most interesting answers to the interview questions:

What did you feel were the main problems with using the Halo system?

The ability to just read small messages on the smartwatch is hindering. Users should be able to read large messages on there as well if they want to.
Users that are stressed (in a hurry, or busy with other things) sometimes simply do not have the time to do a large therapy exercise, thus negating the effect of the exercise.
Therapy exercises should not be repeated in a single day if the user has not explicitly chosen for this.

What did you feel were the main advantages of using the Halo system?

It is very helpful to stop for a minute or two to look at things from a higher point of view for a second. Taking a step back and focussing on a therapy exercise helps to relax.
The system is easy to use since a watch is always on you, as is a mobile phone (usually). This makes it so that the system falls to the background and you don’t think about it any more.
The insights from the web application are very useful for users wanting to better themselves. Especially since users may sometimes not know what caused a stressful moment, or if they feel like something was very stressful while their heart rate shows otherwise. This opportunity to reflect is very helpful in self-improvement.

What are some main points of improvements for the Halo system?

Generate a summary for the user at the end of the day, and push this to the wearable/mobile devices as well.

-Allow for an option to do certain exercises a bit later as well, when more time is available to the user.

Generate short therapy exercises, or exercises that can be easily performed doing other labour during working hours. More time-demanding exercises can be generated outside of these hours.

Activity form

Data from the activity form showed that participants had 3-4 stressful events during the experiment reading, ranging from 2-4 on a 1-5 scale. When compared to the heart rate activity as measured by the system there was about 70% overlap. A relatively large amount of peaks in the recorded data could be related to the perceived drawn heart rate graph. However there were instances in which the recorded peaks differed from the peaks reported the activity form. In one participant’s activity form for instance, one perceived peak was the inverse of the recorded data. In other words, the participant reported that she believed her heart rate was rising, when in fact it was decreasing.

Concluding

This project looked at the possibility to implement a digital anxiety disorder and stress assistant on wearable technology. In the course of the project, the focus was enlarged to also incorporate normal users, rather than patients suffering from anxiety disorders. Detection of high stress levels has shown to be very possible using wearable technology. They can also be detected in time to provide relevant support. The wearable device is able to read sensor data a couple of times per second, which is more than enough to provide real-time support. Wearable technology is also a very useful medium to provide users with therapy exercises.

References

[1] American Psychiatric Association, D., and American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-5. Vol. 5. No. 5. Washington, DC: American psychiatric association, 2013.

[2] Vrijkotte, Tanja GM, Lorenz JP Van Doornen, and Eco JC De Geus. "Effects of work stress on ambulatory blood pressure, heart rate, and heart rate variability." Hypertension 35.4 (2000): 880-886.

[3] Brown, Kirk Warren, and Richard M. Ryan. "The benefits of being present: mindfulness and its role in psychological well-being." Journal of personality and social psychology 84.4 (2003): 822.

[4] Brosschot, Jos F., Eduard Van Dijk, and Julian F. Thayer. "Daily worry is related to low heart rate variability during waking and the subsequent nocturnal sleep period." International journal of psychophysiology 63.1 (2007): 39-47.

[5] Salahuddin, Lizawati, et al. "Ultra short term analysis of heart rate variability for monitoring mental stress in mobile settings." 2007 29th annual international conference of the ieee engineering in medicine and biology society. IEEE, 2007.

[6] Muaremi, Amir, Bert Arnrich, and Gerhard Tröster. "Towards measuring stress with smartphones and wearable devices during workday and sleep." BioNanoScience 3 (2013): 172-183.

\n","readingTime":"24 min read","fullURL":"https://pingen.dev/projects/halo"},{"slug":"virtualad","title":"Virtual Advertising","featured":false,"creationDate":"2015-08-30","category":"Project","excerpt":"Many sports events are broadcast on television with an overlay providing extra advertising space. In this computer vision project, virtual banners were placed inside existing clips of sports events using techniques such as line detection, camera auto-calibration, and perspective transforms.","labels":[{"id":"research","name":"Research"},{"id":"ipcv","name":"Image Processing"}],"cover":"virtualad.jpg","draft":false,"html":"

Introduction

An important aspect of image processing and computer vision is finding vanishing lines for calibration of a camera using these vanishing lines. For this project, we took a look at vanishing lines and in video clips of tennis and soccer games. The aim was to project a banner in the original video clips, transformed so that it looks like it is actually on the field. This means the static image resources need to be updated every frame to account for changes in camera position. We first take a closer look at edge detection techniques.

Edge detection

To detect objects and other interesting variation in an image, we can apply edge detection. There are many methods for this, but most are search-based or zero-crossing based. Search-based methods find edges by calculating edge strength (ie. by using the gradient magnitude), and then searching for local directional maxima of the gradient magnitude. Zero-crossing based methods look for zero-crossings (the point where the sign of a mathematical function changes, from positive to negative or vice versa) in second-order derivatives (ie. the Laplacian) of the image.

Two classic methods zero-crossing based edge detection methods are Marr-Hildreth, and Canny. Marr-Hildreth, the older method, introduces the concept of edge detection with the following process: First, the image is convolved with the Laplacian of the Gaussian. Then to find the edges zero crossings are found in the filtered result. This approach can generate false edges, and depending on your dataset, Canny detection might be a better option. Canny edge detection works as follows: it also first starts with a Gaussian filter, and next, the image's intensity gradient is found. Then, with non-maximum suppresion and hysteresis thresholding the final detection of edges is done. Canny, despite its age, is still state-of-the-art. Not many other edge detectors perform significantly better without large drawbacks.

$\"Figure$

Figure 1. Canny edge detection applied to an image

Strategy

The placement and projection of a virtual advertisement is based on a couple of aspects:

The 3D localization of the side lines of the game with respect to the arbitrarily chosen world coordinate system.
The position and orientation of the camera with respect to this world coordinate system.
The intrinsic parameters of the camera.

Since there were no specific calibration objects in the scene, the court lines of the field, and its known lengths, are used. In computer vision this falls under the umbrella of auto-calibration (self-calibration). See, for instance, Zhang et al. and Orghidan et al.

We chose some sample clips from tennis matches in which the (orthogonal) court lines were sufficiently visible during the panning, rotation, and zooming of the camera. Then, the lines were located using Canny edge detection.

A Hough transform is applied to the resulting binary edge map to obtain an accumulator transform matrix, including theta and rho values for the detected lines. Theta is the angle of that perpendicular vector in degrees ranging from -90 to 90 degrees. The peaks of the transform are then found, and the line segments are retrieved through the edge map and Hough output.

Gap filling and maximum length for line segments are implemented to restrict this selection. Lines are tracked across frames, and the parameters of the detected lines were then used to calibrate the camera. Using the calibration paramters we can then project a virtual banner near a court line. Figure 2 shows this line tracking and advertisement placement for the tennis video.

$\"Figure$

Figure 2. Line tracking and ad placement in a tennis match

We then applied this method to other videos, including one of a football match. Due to the larger variety in camera changes (zooming, large movements) in football matches versus tennis matches, this was a harder case. Figure 3 shows some results.

$\"Figure$

Figure 3. Visualization of placement of 8 banners. The edge map is shown on the left side, and the final image with banner projections is shown on the right side. The top four are vertical banners; the four placed on the field are horizontal banners angled at 60 degrees

References

[1] Z. Zhang, "Camera Calibration", Emerging Topics in Computer Vision, G. Medioni and S. B. Kang, Eds., ed: Prentice Hall Professional Technical Reference, 2004, pp. 4-43.

[2] R. Orghidan, J. Salvi, M. Gordan, and B. Orza, "Camera calibration using two or three vanishing points", Computer Science and Information Systems (FedCSIS), 2012 Federated Conference on, 2012, pp. 123-130.

[3] Beardsley, Paul, and David Murray. "Camera calibration using vanishing points." BMVC92. Springer London, 1992. 416-425.

[4] Canny, John. "A computational approach to edge detection." Pattern Analysis and Machine Intelligence, IEEE Transactions on 6 (1986): 679-698.

[5] Ding, Lijun, and Ardeshir Goshtasby. "On the Canny edge detector." Pattern Recognition 34.3 (2001): 721-725.

[6] Lv, Fengjun, Tao Zhao, and Ram Nevatia. "Self-calibration of a camera from video of a walking human." Pattern Recognition, 2002. Proceedings. 16th International Conference on. Vol. 1. IEEE, 2002.

[7] Marr, David, and Ellen Hildreth. "Theory of edge detection." Proceedings of the Royal Society of London B: Biological Sciences 207.1167 (1980): 187-217.

[8] Sharifi, Mohsen, Mahmoud Fathy, and Maryam Tayefeh Mahmoudi. "A classified and comparative study of edge detection algorithms." Information Technology: Coding and Computing, 2002. Proceedings. International Conference on. IEEE, 2002.

\n","readingTime":"6 min read","fullURL":"https://pingen.dev/projects/virtualad"},{"slug":"grawe","title":"Grawe/Graaf","featured":false,"creationDate":"2015-07-05","category":"Project","excerpt":"Grawe is an aggregated search webpage made to showcase various implementations of (non-)blended search result retrieval. Graaf added to that by making the Dutch part of Wikipedia more accessible to children between ages 9-12.","labels":[{"id":"research","name":"Research"},{"id":"ux","name":"UX"}],"cover":"grawe.jpg","draft":false,"html":"

Introduction

Helping young users search for information on the web is a big challenge for the field of Information Retrieval. Young users of search engines experience difficulty searching for the information they wish to access, especially children aged 12 and lower. More than 60% of children between 9 and 12 years old report needing help from time to time with information acquisition, and this number grows to 90% in children between 5 and 8 years old. Observed shorter average query length together with larger usage of natural language indicate that the use of keywords to generate a specific query is a problem in this target group.

Children's queries have an informational intent, i.e. children query a search engine for information about topics that they assume are available on the web. This in contrast to an adult's queries, which have a more transactional intent. It has been observed that children, in their search with an informational intent, often start reading a result's page from top to finish. They do not exhibit the scanning behaviour that is often seen in adults.

A system that highlights parts of information especially relevant to the child's query can be of use to these children in guiding them to the information they want. Wikipedia is a major source of information for Dutch school children. However, due to a lack of proficiency in scanning a piece of text for relevant information, children will often start to read the Wikipedia article at the top of the page, and continue reading until the information is found. Wikipedia articles contain large portions of text and other content that are not relevant to the specific query, possibly overwhelming a child with information. Children's search behaviour with an informational intent could therefore be improved by providing a system that directs them to those parts of the article's content that are most relevant to the query. The aim of this research was to measure the effects of a new Wikipedia browsing system aimed at children on information retrieval performance.

Grawe

To test this hypothesis a search engine named Grawe (which means 'to dig' in Afrikaans) was developed that aggregated results from a number of different sources (Google, Bing, Youtube, Wikipedia StackOverflow, Tumblr, GitHub, Vimeo, Reddit, Flickr, Nature, and Yummly). These were then ranked and displayed in a list of items based on query relevance. Some features of this basic search engine include a category labeling based on returned results; favicon retrieval; url abbreviation (fully shown on mouseover) for readability; result bundling (ie. 'More results from [domain]..'); and a quick info box for Wikipedia results, recipies, and StackOverflow questions.

$\"Figure$

Figure 1. Left: Overview of the different features in the Grawe search engine on a sample query. Right: Example view of the StackOverflow quickinfo box on a sample query.

Graaf

The Grawe engine was used as a basis for the design of two versions of a Wikipedia browsing system (dubbed Graaf, the Dutch word for 'dig'). The first being almost exactly a copy of Grawe, but only returning Dutch Wikipedia results this time. The other version was the enhanced browsing structure. Upon entering a query, users would immediately be served with a Wikipedia page (eliminating the need to select the best result out of a list, which is reported to be a problem for young users).

A non-standard, enlarged font to increase readability was chosen to display the content. Tables relating to meta-info and auxiliary content such as edit-links are then stripped. Images are kept, but always displayed on the right side of the textual content. The most eye-catching feature is the highlighting feature. The initial query is first stripped of common words (because of children's heavy reliance on natural language in queries), and an iterator function is then called to find all sentences that contain at least 75% of the remaining keywords and highlight those in a yellow hue. An additional synonym highlighting in blue is applied to find and mark synonyms of words commonly used in informational queries. For example 'big', 'large', and 'size' might by synonyms since a user may search for 'how big is the moon', but also 'how large is the moon', or 'what is the size of the moon'. This highlighting was implemented to guide the vision of the user to relevant parts of the text, and\nthe page would automatically jump to the first piece of highlighted text firstly to a highlighted sentence if available, then to a synonym).

The sidebar functions as a scrollbar, with the additional feature of having a visual overview of where in the text you are currently positioned. It was implemented to aid in maintaining the broader picture of an article's structure, as well as being able to see any other highlighted sentences further down on the page more quickly. A minor feature is the horizontal list of suggestions of other articles at the top of the page. This was implemented to provide the user with suggestions of what else to search for, but also to grant some feedback on the search query. Finally, a dictionary was implemented to help users with understanding difficult terms. Every page was parsed by utilizing the Wizenoze API, after which difficult terms could be identified and easier synonyms supplied.

$\"Figure$

Figure 2. Overview of the different features in the Graaf search engine on a sample query.

Experiment

A primary school cooperative was enthousiastic to let their students work with the engine, thus enabling thorough testing of the new browsing system. Children of classes 6, 7, and 8 aged between 9 and 13 were tested using the engine. A list of questions was provided for the children, such as 'What was the name of the volcano that erupted in Iceland in 2008 that disrupted a lot of air trafgfic?'. These questions were designed, together with the school teachers, to be easy enough for the children to know what the topic was about, but not too easy for them to know the answer right away (forcing them to use the search engine to find the answer). Children would choose a random question as a starting position, after which they would have 10 minutes to look up the answer to as many questions as they could do in the given timeframe.

Results indicated that the enhanced structure yielded a 40 second improvement in lookup time (158.9s versus 110.8, N=24, p=0.068) per question. More questions were answered in the same timeframe in condition with the enhanced browsing structure. Accuracy is also better in the enhanced structure condition, as students in that group answered more questions correctly.

$\"Figure$

Figure 3. Left: Mean lookup time per condition, and average amount of questions answered per condition. Right: Answer accuracy for the first two questions respectively.

References

[1] Gossen, T., Low, T., & Nürnberger, A. (2011, July). What are the real differences of children's and adults' web search. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval (pp. 1115-1116). ACM.

[2] Gossen, T., & NüRnberger, A. (2013). Specifics of information retrieval for young users: A survey. Information Processing & Management, 49(4), 739-756.

[3] Duarte Torres, S., Weber, I., & Hiemstra, D. (2014). Analysis of search and browsing behavior of young users on the web. ACM Transactions on the Web (TWEB), 8(2), 7.

\n","readingTime":"7 min read","fullURL":"https://pingen.dev/projects/grawe"},{"slug":"peoplestreams","title":"Tracking People Streams","featured":false,"creationDate":"2015-06-27","category":"Project","excerpt":"During large scale events, e.g. pop concerts or busy places like railwaystaions, large numers of people pass eachother very closely. In order to direct these streams of people or in case of emergency to send help in the quickest way and direct these masses of people to exits, it is important to track such streams and recognize behavioural patterns.","labels":[{"id":"research","name":"Research"},{"id":"ipcv","name":"Image Processing"}],"cover":"peoplestreams.jpg","draft":false,"html":"

Introduction

The management and control of large gatherings of people at events, such as festivals, poses significant challenges to public safety. The high density of people at these events can give rise to dangerous situations. Crowd analysis can aid in mitigating the security risks. In these occasions masses of people are monitored by one or more observers and, if necessary, security services are ready to act and redirect them to different areas. However, given the large scale of these situations, an automatic method to detect potentially dangerous phenomenon can be a helpful instrument to support human supervision. The construction of good crowd models can also aid in the design of public spaces, since these should be able to withstand the crowds that are expected to be there.

People stream analysis

Significant effort has already been directed towards crowd and people stream analysis [3] [4] [5], the detection of dangerous or incongruous events [6] [7], and towards the integration of such analysis into a surveillance system [8]. A 2013 review of video analytics of crowded scenes can be found in published work by M. Thida, et al.

Crowd behaviour patterns

Several behaviour patterns like blockings, bottlenecks, rings, fountainheads and lanes were proposed by Solmaz et al., and also adapted by Stijntjes. In their work, they propose a method of automated analysis of crowd behaviour based on optical flow and regions of interest. This method integrates low-level features retrieved from analysing the optical flow field of a scene, with high level information retrieved through the extraction of features by determining a Jacobian in regions of interest. Since it does not require training, or individual analysis of objects in the scene, it might prove useful for on the fly crowd behaviour analysis.

This project is based primarily on previous research done by Solmaz et al., and Stijntjes. Videos of crowded scenes were studied and patterns which may occur and evolve in risky situations were identified: in particular we focussed our attention on bottlenecks and studied them using a method based on optical flow and eigenvalue maps.

Blockings and bottlenecks may indicate obstructions, such as people stumbling, or partially closed off passageways. Fountainheads and ring formations may indicate that there is a potentially dangerous situation or object at the centre of these patterns. The automated early detection of such events can be a major contribution to health and safety services. Not only to guide the large stream of people to safety orderly, but also to rapidly respond to threats or give first aid.

Data

Footage of crowded scenes is easily found on the web, but most of them are registered from moving cameras or change perspective frequently, making them not quite suited for our purposes. In this project, input obtained from a fixed camera is analysed and predefined patterns (bottlenecks) are extracted where possible. To extend the dataset a number of computer generated video sequences have been created using a free software package Pedestrian Dynamics 2. This allows for data creation in a controlled and qualified manner for analysis. The complete dataset on which we tested our implementation therefore consists of our own generated data and an expansion of footage found on the internet.

Method

For a given video sequence, after preprocessing, the optical flow is determined. A Jacobian matrix is calculated for the entire flow field. From this matrix eigenvalue maps are calculated. Finally, relevant sections denoting possible bottleneck features are determined in the eigenvalue map. Sections marked as possible bottlenecks are compared with a ground-truth in order to determine the correctness of the method. We will go into a little more detail for each of these steps now.

Preprocessing

Raw video sequences are converted to grayscale and exported as individual images. Any detectable bottlenecks in the videos are examined by visual inspection and stored as ground truth. Furthermore, the geometric scale (pixels/m) of the video is determined by finding objects in the scene with a known size; this assumes the video is shot top-down. Angled videos (albeit small angles) were used as well. Finally, the input sequences were scaled to a fixed geometric scale of 10px/m. A typical input image is shown in Figure 1a.

$\"Figure$

Figure 1. Detection method sequence.

Optical Flow

Optical flow calculation is done using the Horn and Schunk method. For every 2 consecutive frames the optical flow is calculated, and the mean is taken of the entire sequence of frames. The result is a 2D flow field of vectors that describes the average motion of a pixel spanning the video sequence, as can be seen in Figure 1b. The flow field is adjusted to the frame rate of the video sequence, and its geometric scale. It is then filtered using a median filter with a kernel size of 1x1 meter (10x10 px) to smooth out the flow field and eliminate large local fluctuations or errors. Finally, the flow field is thresholded to omit small (< 10^-5) values which are assumed to be caused by noise. The flow field is visualized by adding a 360 degree colour mapping based on the vector's angle and size (see Figure 1c) to aid in verifying the correctness of the flow field.

Jacobian and Eigenvalue maps

We construct a Jacobian from the optical flow field by considering a continuous dynamical system $w' = F(w)$ , with $w' = [u(w), v(w)]^T$ and $w(t) = [x(t), y(t)]^T$ denoting particle velocities and positions respectively. Areas in which the stability is assessed, $w^*$ , referred to as critical points, are found by satisfying $F(w^*) = 0$ . When taking into account small disturbances and noise $z = w - w^*$ , and using Taylor's theorem this yields $F(w^*+z) = F(w^*)z + H.O.T.$ . If we then consider only the critical points (satisfying $F(w^*) = 0$ ), and disregard the higher order terms, this leaves $z' = J_{F}(w^*)z$ with $J_{F}$ being the Jacobian matrix:

$\nJ_{F} = \n\\begin{bmatrix} \n\\frac{\\delta u}{\\delta x} & \\frac{\\delta u}{\\delta y} \\ \n\\frac{\\delta v}{\\delta x} & \\frac{\\delta v}{\\delta y}\n\\end{bmatrix}\n$

The Jacobian's eigenvalues, trace ( $\\tau$ ), and determinant ( $\\Delta$ ) are computed and used to label regions to types of behaviours with different colours based on the mapping from Solmaz et al. and Stijntjes as depicted in Figure 1d.

Feature detection

A filter mask is applied to the eigenvalue map to retain only the portions relevant to a bottleneck, according to the values in Figure 2. Colours signifying a bottleneck are yellow and cyan in Figure 1d. The relevant portions are retained in a black and white image as can be seen in Figure 1e. Blob detection is then applied at which point we discard blobs smaller than 15x15px (or 1.5m2). This reduces noise based on the assumption that a bottleneck feature would at least be equal to or larger than this size. Blobs in proximity to one another (within 1m) are merged. The process of blob detection and classification runs in a single iteration, therefore there is no change of blobs converging into a single blob (Figure 1f). Detected blobs are then labelled according to their behaviour type (Figure 1g).

$\"Figure$

Figure 2. Labelling of candidate regions. This table is a corrected version of the one used by Stijntjes.

Results

In comparison to our labelled ground truth data we obtain around 80% precision and around 80% accuracy, as can be seen in Figure 3. These scores are displayed relative to the epsilon value in the filter map from Figure 2.

$\"Figure$

Figure 3. Bottleneck detection result metrics.

Conclusion

We see an increase in accuracy compared to Stijntjes et al. and Solmaz et al. Misclassifications can also be seen to rapidly decline for large vales of $\\epsilon$ .

These results also show that for detecting bottlenecks with the fixed geometric scale a suitable and distinct range of values for $\\epsilon$ can be found empirically.

The method of detecting features using binary filters from the colour labels in the eigenvalue maps is simple but not very robust. Especially on a larger number of types and sizes of features more sophisticated methods (e.g. cluster analysis) are required.

References

[1] Solmaz, Berkan, Brian E. Moore, and Mubarak Shah. "Identifying behaviors in crowd scenes using stability analysis for dynamical systems." Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.10 (2012): 2064-2070.

[2] Stijntjes, Ingo. "Assessment of automated crowd behaviour analysis based on optical flow." (2014).

[3] Ali, Saad, and Mubarak Shah. "A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis." Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 2007.

[4] Santoro, Francesco, et al. "Crowd analysis by using optical flow and density based clustering." Signal Processing Conference, 2010 18th European. IEEE, 2010.

[5] Jin, Xiaogang, et al. "Interactive control of large-crowd navigation in virtual environments using vector fields." IEEE Computer Graphics and Applications 6 (2008): 37-46.

[6] Velastin, Sergio A., Boghos A. Boghossian, and Maria Alicia Vicencio-Silva. "A motion-based image processing system for detecting potentially dangerous situations in underground railway stations." Transportation Research Part C: Emerging Technologies 14.2 (2006): 96-113.

[7] Khan, Sultan Daud, et al. "Detecting dominant motion flows and people counting in high density crowds." (2014).

[8] Fradi, Hajer, and Jean-Luc Dugelay. "Towards crowd density-aware video surveillance applications." Information Fusion 24 (2015): 3-15.

[9] Thida, Myo, et al. "A literature review on video analytics of crowded scenes." Intelligent Multimedia Surveillance. Springer Berlin Heidelberg, 2013. 17-36.

\n","readingTime":"9 min read","fullURL":"https://pingen.dev/projects/peoplestreams"},{"slug":"shard","title":"Shard","featured":false,"creationDate":"2015-05-24","category":"Project","excerpt":"Shard is an Android app that lets you broadcast what you're currently listening to on Spotify to all of your friends live. Listen to your favourite music together!","labels":[{"id":"mobile","name":"Mobile"},{"id":"music","name":"Music"}],"cover":"shard.jpg","draft":false,"html":"

Introduction

You're lying in bed, listening to your favourite tunes, chatting with that special someone living far away. You want to share that moment. Be in sync with one another and connect through music.

Your favourite player is just warming up for the big match. The stakes are high. The tension in the stadium is tangible, pressing even. To get ready he puts on that one track. His track. Your track. You are ready. Connect through music.

These are just two examples of the idea behind Shard. The app lets you broadcast what you're your currently streaming in Spotify to all of your contacts. The status of your player (playing, pausing, skipping ahead) will be sent to all those listening to your feed live. This project was done mainly to play around with the new Spotify Android API and served as a personal introduction into programming for Android.

$\"Figure$

Figure 1. Shard logo.

Overview

The app features a general register/login portal where users fill in profile details such as their name, email address, and phone number. SMS (text message) verification is built in to ensure users fill in correct account details. When they are first logged in, authentication with Spotify is set up through Spotify's web API. This enables the app to access certain Spotify user data needed to function properly. Users require a Premium Spotify subscription to broadcast their status, free users may only listen in on others.

\n\n

The Share tab is where users are able to toggle their broadcasting status to either on or off. When a user starts broadcasting, other users may start listening in on his/her cast. Any track that is playing in Spotify will be broadcasted as well as player status (pausing, restarting, etc.). The user is able to view what is being broadcasted in the player bar (styled similarly to Spotify's player bar) and who are listening.

Discover

The Discover tab is essentialy a list of all contacts also registered with Shard. If a user is broadcasting, a button will become active, allowing them to start listening to that user (or stop listening). Users are able to favourite contacts by pressing that contact for a short period of time. This will place them on a seperate favourites list for easy access. The player bar on the bottom of the screen shows the track that has been received last, together with the player status. A '+' (plus) sign will appear that allows users to add the current track to one of his or her Spotify playlists. Featured casts allow users to listen to broadcasts of users not in their phone contacts.

Stats

This screen shows general user specific stats such as the average and peak number of listeners; a top 3 of listeners; The total time broadcasted and listened; the users' most listened to broadcast; as well as the overall best listened to broadcasts in the app.

Profile

Details such as a user's name, email address, password, or phone number can be edited on this screen. A profile photo can also be uploaded or edited through the user's album or taken directly with the phone's camera.

$\"Figure$

Figure 2. Overview of the app screens. a) Register and login screen. b) Share tab where the user is able to toggle their broadcasting status. c) Discover tab where the user is able to start listening to friends if they are broadcasting. d) Profile editing screen.

\n","readingTime":"4 min read","fullURL":"https://pingen.dev/projects/shard"},{"slug":"soundtag","title":"Soundtag","featured":false,"creationDate":"2015-04-02","category":"Project","excerpt":"Soundtag lets you tag locations with audio messages or music files. Mark your favourite jogging route with mp3 files, leave a romantic message for your lover at work, or remind your roommate to pick up some beer at the supermarket!","labels":[{"id":"web","name":"Web"},{"id":"music","name":"Music"}],"cover":"soundtag.jpg","draft":false,"html":"

Introduction

Soundtag was created as a fun way to reach out to friends, family, and complete strangers, using soundtags: audio files identified by a specific geolocation. When you are near a location marked with a tag, the app will pick it up and start playing the audio file instantly. Travel, search, discover!

The project comprises of a web portal and an accompanying Android app. Users are able to register on either application, but must have an account to login and start using the service. Details such as name, email, and phone number must be filled in.

After logging in, users are directed to the dashboard's map page. This is where a map of all soundtags created by the user can be viewed. New tags can also be created here, as well as changing the home location (initial map view coordinates).

$\"Figure$

Figure 1. Map view. Users are able to view their soundtags on the map; add new tags; and change their home location.

Creating new soundtags is done through a modal dialog. Users pick a position by dragging the map to the desired location (or search for one using the search bar). Audio files up to 10MB can be uploaded to the application. By adjusting the visibility settings, users are able to mark their soundtag as private (only visible to the user itself), or public (visible to anyone). If the public option is chosen, the tag is visible to any Soundtag user by default. If contacts are added, the tag will be visible to only those contacts. Contacts are imported from the Android app and are updated when the user logs in.

Private soundtag use could include reminder yourself to make a phonecall, or a number of tags along your jogging route. You could use public tags to send messages to your friends and loved ones, or complete strangers. By default public tags with no attached contacts can be viewed in the app, but are not played automatically, unless the user enables this setting. Private, and public tags sent to specific contacts will automatically play by default (but this behaviour can also be changed).

$\"Figure$

Figure 2. New soundtag dialog. Users are able to drag the map to a position (or search for it manually), select an audio file to upload to that location, and set visibility preferences.

To give users easier access to all of their soundtags than the map view, the audio view lists all of the tags together with map links, and deletion option. Tags can be played here as well to better identify them, or downloaded to store a precious memory.

Additionally, on the settings page settings such as name, email, phone number, and privacy settings for new sountag default visibility may be adjusted.

$\"Figure$

Figure 3. Left: The audio view lets users quickly browse through all their created tags. Right: General settings can be edited in the settings view.

\n","readingTime":"3 min read","fullURL":"https://pingen.dev/projects/soundtag"},{"slug":"vie","title":"Vie","featured":false,"creationDate":"2015-03-01","category":"Project","excerpt":"Vie is a project that lets users create a collaborative playlist using Spotify. A party host can set up a playlist, and users are able to add songs to the playlist (without owning a Spotify account themselves) through a web portal, and vote their favourite tracks to the top of the playlist.","labels":[{"id":"web","name":"Web"},{"id":"music","name":"Music"}],"cover":"vie.jpg","draft":false,"html":"

Introduction

When you're hosting a party, oftentimes people have music suggestions, both before and during partying. Vie lets you create a collaborative playlist, using Spotify, that allows other people to add and vote for their favourite tracks. Your guests can have a say in what is being played, though the host is always in control.

This project was done to familiarize myself with the Node.js platform and Spotify Apps API. Vie runs as a Node.js Spotify app (note: the Spotify Apps API has since been discontinued), as well as through a web portal. The host starts by creating a new Party in the Spotify app. A random 5 digit Party ID will be generated for the host to display at his/her party. Party-goers can then enter the Party ID on the web portal to login and start adding tracks.

$\"Figure$

Figure 1. Web landing page. Users may get access to the dashboard through their unique party id.

Usage

The Spotify app is meant to function as the party hosts' access panel. Beside creating a new party, a party host is able to login with his/her previously generated party id. An overview of the current playlist, as well as number of votes per track, and tools for deleting, reordering can be found here.

The web portal allows other people to search for tracks using the Spotify Web API without requiring a Spotify account. An overview of the current playlist is shown, as well as options to add tracks to the playlist as well as vote for their favourite tracks. Users may only vote once per track, and tracks are ordered according to number of votes. Of course, the party host is free to override or disable this feature.

Party playlists are automatically removed from the system after a period of nonuse, unless the host opts to store the playlist for future use. Tracks may also be stored to a seperate playlist to store for another time.

$\"Figure$

Figure 2. On the dashboard page users are able to search for music in the entire Spotify library through the web API, and add them to the queue; vote for their favourite tracks; or remove tracks from the list.

\n","readingTime":"3 min read","fullURL":"https://pingen.dev/projects/vie"}]

Introduction

Sustainable Computing

Measuring power consumption

From power consumption to carbon emissions

Tools and Ecosystems

Experiments

References

Introduction

Background

Related Work

System Overview

Use-case

Causal reasoning

Challenges & Lessons learned

Future Work

References

Introduction

Federated Cloud

Multi-cluster topologies

Technologies

GAIA-X

Edge environments

Security

Experimentation

Conclusion and Future work

Further reading

References

Introduction

Related work

Remote sensing

Learning-based remote sensing

Data

Methods

HOTMAP

SVM

DNN

Results

CA & HOTMAP

SVM & MLP

DNN

Concluding

References

Introduction

Related work

Colour indices

Neural Networks

U-Nets

Data

Method

Results

Estimation

Limitations

References

Introduction

Research

Data

Implementation

Initial ranking

Relevance feedback and reranking

Supplementary detectors

Method

Results

Concluding

Further reading

References

Introduction

Competitive StarCraft

Research

Bot Overview

Features

Data

Strategy Tree

Strategy Recognition

Strategy Generation

References

Introduction

Research

Robot platform

Hardware

Software

Geert Pingen @gpgn