How Open Source Tech is Shaping New Development

It is commonly known how fast the tech industry changes. From consistent function deprecation to emerging libraries and frameworks it is easy to get lost amongst the constant iteration of standards. Taking a look back a mere 4 years ago shows a tech landscape centered with wider adoption of closed technology and proprietary frameworks. In 2018 it is unbelievable what you can do with so little in terms of software development. Modern platforms as a service make it easier than ever to deploy applications straight from GitHub to the cloud and make use of continuous integration with a click of a button. This has caused an exciting change in the tech industry with regular developers having free access to deploy and test their applications without having the complications of configuring server architecture or provisioning resources.

So why is open source shaping new development? Well, it comes down to multiple points. Aspiring and experienced developers have more access than ever to professional code bases where they can contribute and integrate themselves in a project or technology developed by thousands. You notice a bug in Visual Studio Code? No problem! Feel free to fix it and share it with the millions of users across the globe. Doing this adds merit to yourself as a developer making it good for resumes, learning, and community involvement. In Open Source a student and senior developer are on the same playing field. They both have access to the same resources, platform, and documents that support the code base which makes the dynamic of contributing to a project freely so interesting. What is even more surprising is the adoption of this development model by development firms and pure tech companies. For example, appbase.io is a mid-sized company specializing as a platform as a service for elasticsearch and NoSQL db storage. I did some open source work with them a few months ago by fixing a few bugs and small feature enhancements. What’s interesting about this company model when it comes to Open Source is that by allowing developers access to the codebase and making software issues public is that it allows a dev to dev tradeoff. Appbase.io gets potential free debugging and feature enhancement for its paid service and technology and the open source contributor gets access to a professional codebase and a line of contact to senior developers who in most cases will personally guide contributors to fix or enhance a feature. This happens in a wide variety of projects from Google Chrome, Mozilla Thimble, Blockchain (loads), and many more. A lot of applications used every day are developed and maintained by ordinary people around the world which makes the prospects of being a developer so broad and essentially having the sky as the limit. People tend to apply themselves more when doing something they enjoy and open source allows not just developers but people in all parts of the tech industry to specialize and contribute to projects and technology that truly inspire and drive themselves as a professional.

Overall I am extremely excited to see the how Open Source will grow and shape new and old applications. I believe a hybrid approach of profitability and having an open codebase is entirely possible and should be encouraged with emerging technologies. Open Source is proven to create a higher adoption rate (just take a look at TensorFlow, Blockchain, etc)  and kickstart development for new projects. I hope Open Source will continue to bring new and exciting technologies to the revolutionize the tech industry.

 

Advertisements

Release 0.6 – The Finale but not the end

Over the past eight months, I discovered a community in which I never thought existed. A community full of friendship, challenge, and hardship. I learned how to put my mind to the test and collaborate with individuals all over the world to reach one common goal. Improve and develop great software.  I’ve contributed to a variety of organizations like Mozilla, MoePlayer, Chart.js, and many more. I feel my contributions to the open source community have made excel as a software developer but tenfold as a problem solver. A lot of open source work is implementing efficient and unique problem-solving methodologies. You have to figure out how to not tackle a problem from one angle but several. You need to know how to dissect an issue to its barebone functionality and have the ambition to never give up even when the issue you are tackling is tough and shows no sign of remorse. This is what makes an individual an excellent open source developer and problem solver.

One thing to realize with open source is that you don’t know everything and it is likely that you don’t know nearly as much as you did about software or various coding languages. There will be projects that look like alien code even though you’ve been coding that language in college or university for three or fours years. This is when you discover the contrast of practical production code and the “simulated code” that you come across in coding classes that only aim to teach you a single or few concepts. It is important as a software developer to have exposure to real-world code as much as possible because this is what will prepare you for the workforce and future opportunities. Of course this not to say that simulated learning code provided by professors is bad and I do agree that it is crucial to help students learn the core concepts and application of different programming languages. I am rather stressing that some students tend to get into a cycle of being spoon fed code and explicit instructions how to complete a certain task. This is bad because in the workforce and in my personal work experience there is no project spec or sample code that dictates exactly how to accomplish the project. At my prior job I was simply told what was needed and had to design, develop, and test the software myself. So how does this all tie in with open source? Well, open source provides the means to work on real-world code with industry leading developers and gain valuable experience that you can place on your resume or use to better yourself as a developer. Taking this all into regard I decided to tackle a hard issue for my last release.

The issue I decided to tackle for this release was to add a calendar widget to the date filter for dejavu. In plain text it seems very simple to implement however I have very little experience with ReactJS and haven’t dived into state management and passing data from different parent and child components whilst using an external date API to return the correct data as well as conform to existing functionality in the project. Overall it seemed like a great way to challenge myself and I’ll walk through my development process. Luckily to start a maintainer of the project gave some good info on where to start as well as a good date picker API that fits perfectly with react called react-datetime. I began reading over the component that handles filter input called “SingleMenuItem.” Currently, in this component, we are using events to grab and validate text input instead of having the filter submit like an HTML form for example. This was a bit confusing for me and from my understanding, the input data is being stored in the components state and then accessed from a parent class called “FilterDropdown” which contains several “SingleMenuItem” components. The problem with this approach for me was having the react date picker conform to our existing functionality in the project which was very event focused. But as one knows from using a date picker widget most of the time you simply click on the date and it fills the field. Knowing this myself I assumed that the onClick() event could be used to trigger our existing input event functionality and decided to go down that route initially.

Example of dejavu events:

valChange = (e) => {
    var filterValue = e.currentTarget.value;
    this.setState({
        filterValue: filterValue
    });
    this.props.getFilterVal(filterValue);
};
// Further down in render()
 input id="{keyInput}" type="text" placeholder="enter value" onKeyUp={this.valChange}

(Chevrons removed to stop WordPress from processing input tag literally)

Essentially like shown in the example above I had to implement something similar but have it work for the react date time component. Now the date time component has an attribute call inputProps which can be used to dictate custom input properties for the date picker. So my first attempt was to add the onClick() event to the inputProps attribute but as a result, this canceled any onClick() event defined in the Datetime API thus making it run valChange with an undefined event. I ended trying almost every event that is offered in React and had countless hours of half-baked implementation where the user does some abstract event to trigger the function to grab the input value. I did consult one of the maintainers for the project about this issue and he was very helpful but I think the main problem with all the hours of testing and coding was my lack of experience with React applications.

Fast forward to my current implementation I needed to add three additional functions to SingleMenuItem.js because the  tag handles events different than a regular input (API reference) and returns date data in milliseconds (UNIX time) depending on the event which in this case is onBlur. So once onBlur is triggered we attempt to parse the date and set the filterValue accordingly. This logic happens in setDate. For range values, it is set a bit differently. I created two functions called setRangeStartand setRangeEnd which handle this use case. Depending on which one is set first (starting date or ending date) the function will either store the date in the dateStore state to be used by the contrasting function that will set the dates as the date range for filterValue.

—Code reference—

This is the implementation I decided to create a pull request with because I believed that it would work for all use cases. To my surprise, a maintainer got back to me and indicated that the date picker has to adapt to the date format of the data and thus be created dynamically basing its format on the data. This created a HUGE hurdle for me because it involved accessing information dynamically across objects that don’t have any relation. I have since replied to the maintainer asking if there is a way to access the elasticsearch data within a component that has no relation. I figure if I can access just one date I can base the format off that and create the date picker, however, the challenge is more accomplishing this in React with entry-level experience. I hope perform a quick update with the fix but overall I’ve enjoyed the challenge this issue has given me!

Release 0.5 – Database as a service

Over the past few weeks, I learned a lot outside of my existing knowledge of relational databases like Oracle, MySql, PostgreSQL, etc. I decided to work on Open Source project Dejavu which is a Web UI for elasticsearch allowing users to get a visual representation of data as well as import data from JSON and CSV files into their databases or documents. The project is created by a company called Appbase.io which offers a NoSQL database as a service in the cloud. Dejavu is just one of many software applications offered when you sign up with Appbase.io. So far my experience with the developers has been great and I already successfully fixed two issues and had the pull requests merged.

The first issue was fairly interesting. Basically, when you are applying filters to data in dejavu the drop-down UI doesn’t go away after it has been applied. Dejavu is written in ReactJs so I had to learn my way around and code read for an hour or so just to get the gist of the code structure and how it is applied in the browser. I found it really interesting how you can set object states in React and also creates tags that act as HTML components. For example, the dropdown is coded as it’s own tag which in itself is a composition of HTML/CSS to make that dropdown. This makes it convenient because you can simply write <dropdown> instead of all the HTML that makes up the dropdown for the project. To tackle this bug I initially started playing around with the state of the dropdown and creating my own custom state to check if the dropdown has been hidden or not. The main problem I was having with this is that all the dropdowns are rendered when the DOM is loaded so changing the state of dropdown to hidden or shown would make all of them hidden or shown which created a problem. Luckily within the time, I was fixing this bug a developer for dejavu replied to the issue saying that he too encountered a similar issue and that a simple fix was to simulate a click outside of the dropdown thus closing it.

As a result, I implemented divyanshu’s suggestion:

applyFilter = () => {
    if (this.state.filterField != null && this.state.filterValue != null && this.state.filterValue != '') {
        this.props.filterInfo.applyFilter(this.props.type, this.props.columnField, this.state.filterField, this.state.filterValue, this.props.analyzed);
        document.body.click();
    }
};

The code above shows the function “applyFilter” that is called when the user clicks apply on the dropdown by adding document.body.click(); at the end of this function, the dropdown is automatically closed without user interaction. After verifying and testing the solution I submitted a pull request which got merged faster than expected.

Since fixing this issue went so smoothly I decided to take up another for this release. The second issue was with the export UI modal being closed when clicking outside of the modal on the overlay. 

I found this issue to be a just as interesting as the first for I have dealt with similar issues in my personal projects that use an overlay with a div. Now knowing how the project is structured I immediately found where the issue was occurring and started debugging. Dejavu’s project file has the best naming and structure that I’ve seen in an open source project and appropriately the code for handling exported JSON has contained in “ExportJson.js.” While debugging I found that the close function which closes the export modal was called in a very general way which essentially made it close the modal if anywhere in the div was clicked which unfortunately, in this case, means the overlay as well. To fix this I got to play with the one thing I wanted to in React which is state! I was so excited to apply a state fix to this issue and I did so by creating a boolean state called “exportClose.” I used jQueries click() function in conjunction with the state “exportClose” to only set it to true when the “x” button is pressed on the export modal (whose class is appropriately called “.close”). Then if the exportClose state is set to true we close the exportModal which changes another state called “showModal” to false.

close = () => {
    var thisRef = this;
    $(".close").click(function(e) {
        thisRef.state.exportClose = true;
    });

    if(this.state.exportClose) {
        this.setState({
            showModal: false
        });
    }
};

In contrast to the close function, the open() function has “exportClose” being set back to false when the export modal is reopened by the user. After some testing and fine tuning this ended up being the final solution for the issue and I submitted a pull request with my code. To my surprise yet again the pull request got merged within no time and I successfully had two bug fixes completed for a project I didn’t really think I could get any in initially. Overall I hope to continue working on dejavu as well as explore the other data based projects that Appbase.io offers. It has been a great time learning new approaches to data storage and querying as well as playing around with ReactJS!

The Open Source Comeback – Data Projects

Over the past three or four months I’ve had a lot of ups and downs with working on open source projects. From having big fixes and features rejected all the way to stranded pull requests never being merged. This, of course, can be disheartening and has caused a lack of motivation to continue open source development. I feel overall open source, in general, is the ideology of free, collaborative, and open code as well as an open environment. What I mean by an open environment is the ability for the average software developer to jump onto a project and receive a warm welcome, support, feedback, etc. For those who have done a  substantial amount of work in open source definitely know the difference between a project where the developers are engaging, helpful, and kind compared to a project where the developers are standoffish, exclusive, and not too fun to work with. These traits heavily depend on the type of project and the individual’s interaction with the maintainers of the project. For me personally, I was growing tired of the broken communication, stranded PRs, and messy project organization that I didn’t feel like contributing all that much however I am excited to say that has changed since switching to a new project.

In preparation for my job after graduation, I decided to look for data-based projects because my work will be primarily be done supplying data solutions to clients. Big data technologies like Apache Hadoop, NoSQL, and ElasticSearch were all things I’ve heard of before but never took a big look at trying to develop for via open source. It took a while to weed out what projects I could realistically get a foothold in and came across one really interesting one called Dejavu.  Dejavu is a web UI for Elasticsearch which is a hugely popular open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Elasticsearch is extremely fast by using distributed inverted indices which allows for quick matches from very large data sets. Elasticsearch is primarily an API and what dejavu provides is the user interface to visualize and query the data. To use dejavu you need a NoSQL database that contains your data. This is provided by the same company “appbase.io” that created dejavu. Appbase.io is a hosted Elasticsearch service with built-in publish/subscribe support for streaming document updates and query results.

Appbase.io Dashboard: 

Simple Dejavu Example: 

This has been a huge learning curve for me to understand all the different data querying and big data technologies, however, I feel like this will benefit me tenfold when I migrate to the workplace and have a general idea and experience working with data tech that is outside of the RDBMS spectrum which I am so familiar with. So far the developers at Appbase.io have been extremely welcoming and I’ve already begun working on some issues for dejavu that I will write about in more detail for my next release. At the core, dejavu is built with ReactJS which is really interesting because I haven’t developed with ReactJS before so it’s been a lot of fun figuring out and understanding the capability ReactJS holds. I’ve also had to learn how to use the Elasticsearch and appbase.io javascript API so I can create the appropriate test data to debug the issues I’m working on which has been great so far. Overall I’m excited about this project and hope I can work on it a lot more and expand into different areas of appbase.io.

Release 0.4

After fixing a couple bugs in Chart.js I began to migrate to the bugs that I possibly couldn’t 100% fix but that rather made me interested in why this issue occurs. Like most things in life when people are interested in an activity they tend to apply themselves more when compared to others. For me, I can work both ways, however, I do enjoy working projects and issues that affect real-world application and a lot of my new bug fixes and code reflect that.

I decided to take on two bugs for this release, the first bug allows too many data points to be hovered with the tooltip on certain charts. Example: 

Luckily when I asked to work on this issue, user etimberg helped point me in the right direction and said to take a look at the code in “core.interaction.js” to solve the issue. After hours of investigation and debugging I started drawing various conclusions as to why this bug occurs. I narrowed it down to one function simply called “x.”

x: function(chart, e, options) {
   var position = getRelativePosition(e, chart);
   var items = [];
   var intersectsItem = false;

   parseVisibleItems(chart, function(element) {
      if (element.inXRange(position.x)) {
         items.push(element);
      }

      if (element.inRange(position.x, position.y)) {
         intersectsItem = true;
      }
   });

   if (options.intersect && !intersectsItem) {
      items = [];
   }
   return items;
}

In summary, this function retrieves the tooltips in the range of “x” from function “parseVisibleItems” and then pushes the tooltip item(s) into the items array. I originally thought this bug was located in “parseVisibleItems” but after debugging further and further, I found out that “parseVisibleItems” returns all the tooltips, in general, leaving “element.inXRange(position.x)” to deal with parsing those items returned only if they are in the range of X. The keywords here that are fundamental to the bug is “inXRange.” Naturally, with a small number of data points there would be only one point in the range of X however in the case we are dealing with there upwards of three hundred and fifty data points for “x” meaning that there could be several points that are in the range of “x.”

I believe the philosophy for “inXRange” is that for a user it makes it easier to highlight and hover over data points without having to be precisely on the x coordinate in which the data point is located. “inXRange” allows the user to be somewhat close to the data point and still have it highlighted and a tooltip showing the data. But still like mentioned above when we have excessive data points close together, multiple are highlighted and returned with the items array.

Now how do we go about fixing this issue? Firstly I had to find the threshold where the multiple highlighting of data points occurs. After testing I found that “inXRange” will return more than one tooltip item as the data points increase past 40 (Here is an example of 35 data points). 

Now when we have more than 40 data points it essentially requires a tighter range for x because we have the data points closer and closer together. What I ended doing is making a copy and modifying function “inXRange” so that the range will be smaller allowing for only one data point to be returned. Example code:

x: function(chart, e, options) {
   var position = getRelativePosition(e, chart);
   var items = [];
   var intersectsItem = false;

   parseVisibleItems(chart, function(element) {
      if (element.datasetlength < 40) {
         if (element.inXRange(position.x)) {
            items.push(element);
         }
      } else if (element.datasetlength >= 40) {
           if (element.inXRangeSmall(position.x)) {
              items.push(element);
              // Cleanup extra elements
              if (items.length > 1) {
                 items.splice(1, items.length);
              }
           }
      }
      if (element.inRange(position.x, position.y)) {
         intersectsItem = true;
      }
   });

   if (options.intersect && !intersectsItem) {
      items = [];
   }
   return items;
}

In summary, this code applies my modified “inXRange” function when the datasetLength (amount of data points) is greater than 40. I then added an additional piece of code to splice any extra items in the array in the event that a certain datasetLength returns more than one tooltip item in the range of x. This code produces the following result:

As shown above, since the range of x has been shrunken only the correct tooltips are being returned. This is currently what I have in for my pull request and I am waiting for a response from the Chart.js maintainers.

For my second bug I decided to investigate an issue in which charts aren’t showing when they are nested under the <details> tag in a div. I found this bug to be very interesting because it only occurs in Firefox and seems to be a problem with DOM loading. Example:

(Left: Firefox   Right: Chrome)

I began to dig around with the inspector tool on Firefox to see if the canvas was even showing. It turns out it was, however, there is a render time to produce a chart and when the DOM is loaded the chart isn’t loaded instantly at the same time making it not appear. The way to we get around this in Chart.js is by triggering a CSS animation when the chart is initialized so that the DOM recognizes a change in state and shows the canvas. However, Firefox cannot detect the CSS animations thus can’t refresh the DOM with the rendered canvas. A manual way around this is triggering the animations yourself with the inspector tool. Example: 

I began doing some research to see how other projects and developers perform DOM node insertion detection and came across an API called MutationObserver. This is a fairly new API however it is supported by all modern web browsers which would make it great for implementation with Chart.js. Basically, a MutationObserver provides developers with a way to react to changes in a DOM. It was made to replace the now deprecated “DOMContentLoaded.” I tried out some small examples on how to add and use a MutationObserver in a project but I thought I’d better ask the main Chart.js developers if this is a worthwhile solution to develop.

Unfortunately, a MutationObserver wasn’t the right API for our particular project, however, I still feel like there is a way to fix this on our end or possibly figure out why just firefox can’t detect our current implementation. Overall I enjoyed working on these two bugs and hope to accomplish more in the future. I may look at joining another project temporarily as I wait for my pull requests to be reviewed.

 

Chart.js Project Status Update

I’ve been working on Chart.js for a few months now and my experience has been excellent so far with almost completing now my fourth bug fix for the project. I have recently made a pull request to fix a bug where too many data points are being hovered by the tooltip in certain states. Example: 

I’ll go into more detail about how I approached and began fixing this issue in my Release 0.3 post which will be coming later this week. Overall I enjoyed working with this bug and can’t wait to get some feedback which brings me to my main point of this post…correspondence. I currently have two PRs in for Chart.js, one which fixes the tooltip showing off screen for certain graphs and now my most recent one fixing the tooltip highlighting too many points on the graph.

With the image shown above, I can’t help but wonder what state the projects in and if we’re in a state where development is coming to a halt or if this is a temporary break taken by the maintainers who possibly have other issues or projects to tend to. While I was browsing the other PRs made by other contributors they seem to have some good quality to them however they require a review to see if they can be correctly merged or not. Chart.js, in my opinion, is not completely done with development and free from errors, I feel there is a lot that has to still be done which is clearly shown with the four hundred and eight open issues that range from bugs, enhancements, test cases, and questions. Though I’m not 100% sure because each open source project is different I’m leaning towards the cause of so much pull requests being abandoned is due to the fact that there aren’t enough maintainers to handle a project at this scale. Right now there are seven maintainers and only two of which that I’ve actively seen comment and review pull requests. The two maintainers I have interacted and collaborated with both have their own personal jobs in different parts of the world which could make it hard to tend to a project that doesn’t pay or have too much incentive to excel in other than the ideology that open source provides to the software and tech community.

In the meantime, while I wait for a review and possible merge of my current pull requests I will still continue to work on a couple more bugs as a part of Chart.js because I feel like this is just a temporary lull for the project rather than a permanent downturn in productivity.

Release 0.3

Open source development across GitHub and other platforms tends to be a cyclical process. One takes a bug or feature request, debugs and trys to implement what has been outlined, and then creates a pull request with the fix. Of course, as a developer, you can find your own bugs and fix those or post them as an issue for others to view and fix but I feel overall you end up doing the same thing which is not exactly a bad. Open source provides an existential amount of real-world experience working on huge industry leading projects some of which I have added to my own personal resume. However, after being so involved in the open source community and putting out three or four substantial pull requests for each release, I decided to take it a bit easier on constantly fixing bugs. It took a lot of my time to fix some of the bugs I went after and was mentally draining to be trying to fix or create something constantly. I don’t regret spending most of my time fixing bugs however for this release it was hard to jump in with the same energy as I did previously.

Does this mean future releases will be small? No. This release is small purely out of choice, mental break, and the fact that I’ve been developing other private repos on GitHub for another project for school. With two months left in my college career, I’ve also been focusing on relocating for work and learning new skills for my job upon graduation which has taken up a lot of my time as well (it’s hard to code and travel to look at apartments on the weekend). Enough of the excuses and reasons lets jump into where I’m at with  Release 0.3!

For this release, I decided to take on an interesting bug in Chart.js. I found this bug to me almost one of the most interesting ones I’ve seen.

Example: 

The main problem here is that the tooltip, which is the black box containing the data is going off the screen and following the data as if it is continuing. It’s weird because in this example it’s only supposed to print the last four hundred days however its actually printing more than the minimum value than where the data is initially supposed to start at. After analyzing the issue I initially thought it was a data problem since its not sticking to the minimum value assigned but one of the maintainers steered me away from investigating further into that idea.

I began investigating the file in the project that contains all the tooltip code called core.tooltip.js. Where I added the following logic.

var border = active[0]._xScale.left + 7; // 7 used for graph offset
if (existingModel.x > border) {
    for (i = 0, len = active.length; i < len; ++i) {
        tooltipItems.push(createTooltipItem(active[i]));
    }
}

The problem with this logic is that it is not secure and breaks several tests associated with the project. Chart.Js can produce a variety of different charts, for example line graphs, pie charts, and bubble charts. The implementation I originally provided only works for a line graph and doesn’t account for the difference of attributes in different graph types. The result of having this implementation was several errors in the build test for different charts.

After revaluating core.tooltip.js I found the code that deals with the mouse event and tried to implement what I had previously did in the update() function. I ended up having to make a global variable that retrieved the chart area so I could use it to find the boundaries of the chart. For example:

var globalChartArea;
...
globalChartArea = chartArea.left;
...
if (e.type === 'mouseout') {
    me._active = [];
}
else if(e.x > globalChartArea) {
    me._active = me._chart.getElementsAtEventForMode(e, options.mode, options);
}

Developing this block of code was mainly the product of not knowing what variables can access certain objects which was later clarified by one of the lead developers of the project.

With this and more feedback from the Chart.js developers I created what I thought would be my final commit to this PR. Result: 

var border = me._chart.chartArea;
me._lastActive = me._lastActive || [];
if (e.type === 'mouseout') {
   me._active = [];
} else if (e.x > border.left && e.x < border.right) {
   me._active = me._chart.getElementsAtEventForMode(e, options.mode, options);
}

To my surprise there was small bug in the code outlined by developer Simon Brunel that I hadn’t noticed before.

Simon also noted that we need updated unit tests for this fix which was an added task I had to accomplish. Taking Simon’s other code suggestions I sent a commit which would end up being the final fix for this issue. Result: 

var area = me._chart.chartArea;
if (e.type === 'mouseout' || e.x < area.left || e.x > area.right || e.y < area.top || e.y > area.bottom) {
     me._active = [];
} else {
    me._active = me._chart.getElementsAtEventForMode(e, options.mode, options);
}

This code fixed the issue Simon outlined leaving only the unit tests that had to be updated. The main problem with the unit test for this fix is that currently it only accounted for the mouse position of x and not y. To fix this I added the implementation used to get the x coordinates and modified it so it would work to collect the y coordination of the mouse. This seemed to fix the problem with the unit tests and got approved by one of the project developers Etimberg.

Now it is mainly waiting for a merge or request of changes on the fix. Overall I actually had a lot fun with this bug, it is something I was really interested in fixing and enjoyed the correspondence and collaboration with the other Chart.js developers. I plan for my next release to be a fair bit larger however the fix for this bug wasn’t as simple as I thought and was a great learning experience.