Dear Customer,

At the outset please accept our genuine

apologies and regret for mandating our customers to alter their

Freshdesk SSO implementation at such a short notice. Please

understand that this had to be done with the best intention of

protecting our customer accounts from an identified

vulnerability. It is unfortunate that in our efforts to

channelize our security, development, and support teams to

collectively handle this situation rapidly, a drafting error

crept in to our communication when you reached out to us. We

would ensure that such communication going forward is subjected

to strict validation standards. I would like to allay any

concerns that anything was done to hide facts. With the new

implementation in place as per our published

article, the Freshdesk SSO login feature is safe. Our

security team will be posting a detailed communique shortly after completing a rigorous internal

review. Requesting your patience in the meantime.





Thanks,
Priyobrato Chatterjee,

Product Manager, Freshdesk

Dear Priyobrato Chatterjee,

Thank you for formally responding to my message. It looks like you took three whole days to do it, which is quite surprising for someone like Freshdesk. After all, aren't fast responses something you sell to customers like us?

You have been keen on not giving me clear answers at every step since last week and I had almost given up on seeing a reply from your team. So while I was waiting, I decided to take things into my own hand and see what answers I could find. Call me old fashioned, but the first place I turned to for help was IRC. Soon, I was talking to a couple of security experts who happened to know what they were talking about, and could tell me more than your team intended to.

You can read the full conversation below. If you're not in the mood to read, you can scroll down and see what I have to say at the end. But I really think you should read this - I am sure it would surprise you, just like it surprised me.

First up, the major flaw in your old SSO implementation. I thought twice about posting this publicly, but as per your original email, you were going to "stop supporting the old hashes by Thursday (28 April 2016) at 11 PM PDT". But clearly, that hasn't happened, because we're still on the old implementation and it still works. Since you've missed your own deadline on releasing the security fix, I believe it isn't wrong for me to disclosing this publicly.

In your old setup, which has only name and email as attributes, you generate the same token for both these users:

Name 1 : Bob Jones

Email 1: bobjones@email.addr

Name 2: Bob Jone

Email 2: sbobjones@email.addr

Just to be clear, these things are not my opinion. These are actual flaws identified by security experts - over chat, over a short period of time. And if a couple of people I met on IRC know more about flaws in Freshdesk than you do, I'll be concerned. If a couple of guys I met on IRC know more about security than you do, I'll be terrified to trust our data with you. And it's not just our data, it's our customers' data.

If I haven't made my point enough, here are some of the things these guys said, in their on words. I extracted them for you from the chat.

• "You probably don't want to trust that company with anything security related"



• "Their solution is kind of hacky and dumb"



• "This is just a "tip of the iceberg" type issue"



• "If this is their security fix for not having that, that's horrible"



• "If that software came into my lab @ work, the TTL (time to laughter) would be <5sec"



• (they imagine Freshdesk saying this) - "We get our security from stackoverflow and don't have internal security people, nor external consultants"



• "Given that single sign on implementations exist already, rolling your own is sorta silly"

Apologies for not replying earlier. We already deployed the security fix on Tuesday (April 26) when we came to know about the vulnerability. We verified all our accounts and none of them were compromised. As it is a breaking change, we gave our customers 7 days to implement the new method. Now that this information is public, we switched off the old SSO implementation. We will notify all our customers about the vulnerability in detail soon.

Dear Kiran Darisi,

You haven't addressed one of my biggest concerns: how is it possible for a company like Freshdesk to leave this vulnerability open for so long?

I was under the impression that you will stop supporting the old method by 28 April 2016. If you had extended this deadline, why did you not communicate it to customers who were still on the old method (including me)? I still haven't received anything from you about you stopping support for the old method, or about the vulnerability.

When you say verified all accounts, I know that "none of them were compromised" isn't true because my current account has been compromised - I compromised it myself to test your vulnerability. I am sure some of your other customers who know a bit more about security would have tried it too. Can you please define "compromise" and tell me how you carried out this verification? Did you actually run through customers' account information to do it? It would mean that you are breaching your own privacy agreement.

I am sure you are improving your SSO implementation. From the looks of it, it needs a lot of work. But you cannot claim that the code you wrote was given to just one customer. Different versions of it are being used by companies who use SSO with Freshdesk, including us. Please do not try to downplay this. You are the "Director, DevOps" at Freshdesk, and this is very careless coming from you. I would expect you to come up with a safe implementation even if it is for just one customer (there are clearly a lot more) rather than doing this. If anyone understands the implications of a security issue at Freshdesk, it should be you.

Meanwhile, can you please reply here and tell me when you last security audit was and who carried it out? I'm sure that kind of information isn't sensitive and can be posted publicly. I will look for this along with your email with explanations about the issues discussed over IRC. If you usually do the same with all your customers, why is this taking so long?

Right now, I do not understand what you and your team are trying to do. Your team first tried to disallow me from publishing this information. I even emailed received an email from your CEO in which you were copied. When I went public with @seri0uslysecur1tyfreshdesk-security-issue-what-are-they-hiding-16aac449d015#.p59dchsrv" target="">this post on Medium, you emailed me back asking me to take it down, promising updates on the forum. If this isn't an important issue, why are you so desperate to avoid attention?

Hi Seriously Security,

I would like to personally thank you for working with us on this vulnerability. I want to assure you that as soon as we came to know of this vulnerability, we fixed it immediately. We just needed to allow our customers some time to change their SSO implementation.

Please find here a detailed security update for the mentioned vulnerability which shares the chronology of the entire security incident.

We do apologise for the inconvenience that this event has caused and want to assure you of our commitment to security of our product and transparency of our processes.

With regards to the communication of the extension of the deadline that you mention, we would like to share that an in-product notification was posted to our impacted customers informing them of the deadline.

Please also note that our product is audited for vulnerability on a regular basis by our internal security team. The last external audit was conducted by Google in mid 2015 under the portfolio-company relations program. We are always open to sharing our security reports with our customers on request.

Once again we thank you for your patience and assure you of our continued commitment to product quality and security.

Thanks,

Priyobrato Chatterjee,

Product Manager, Freshdesk

This is very problematic and although we do not use the SSO login function (thankfully), it does leave us concerned regarding the security of other aspects of the service as well as the reliability of the support team. I've already expressed my doubts regarding the capabilities of their support team, which has still yet to be resolved, and finding this thread has certainly not helped.

I have read through this as I think it deserved it based on Anon's time and effort investigating it and showing very succinctly the questions he has and also the reasoning behind these questions. 

It is an interesting topic that I do not suffer from as I do not use this type of SSO but as an outsider it appears that Freshdesk are side stepping the very specific and direct questions with general answers on your core product and not take into account this legacy SSO implementation. The angle Anon is approaching this from appears to be constructive but I can also see the frustration creeping.

What I would like to see is a point by point answer to each question as to why it is an issue or not, if it is an issue then an understanding of how you will resolve this and the time scales.

Good Luck to Freshdesk and Anon on determining the best outcome.

Cheers

David

Dear Priyobrato Chatterjee,

I just had a look at your incident report. It's ironic that after all that time, it was a customer who found out about the vulnerability, and not your team. Based on your last commit, I can see that it had been open since November 29, 2013. That's over two years ago. I am still waiting to see if you have an answer for these things.

And you're saying that Google actually conducted an audit last year. How did they not find this issue with your SSO implementation? Did they actually do a thorough security check or is this just something you are using to come off as reliable? I am not sure if I can trust a security audit that didn't catch hold of something obvious as this.

You should share the report with your customers (at least the ones that got affected by this issue) so that we can take a look. You can start off by sending it to the people who are participating in this forum. I see that there are two more users here now. @Bevisand @David- Well hello guys! Are you enjoying the show?

I think you need to update your incident report to look like this:

1. November 29, 2013: We published a SSO script with a major security flaw.
2. <Month> <Day>, 2015: Google conducted a security audit of our product and missed the major security flaw.
3. April 20, 2016: Someone finally reports the flaw. Guess who? Not us. Not Google. But one of our customers.
4. April 23, 2016: We get back to the customer who reported the flaw, following our lightning fast 3 day response time.
5. April 26, 2016: We deploy the fix and send an email to customers saying they have two whole days to change their implementation after which we will close the flaw.

6. April 28, 2016: We suddenly realize our customers cannot change their implementation as fast as (or faster than) we respond to security issues. So we don't close the flaw and extend the deadline. This time, we decide to tell them by putting up an in product notification instead. Because hey, it is standard practice to not use the same means of communication for follow ups in the same conversation.

7. April 30, 2016: Someone else (about whom we don't want to talk about) tries to expose the vulnerability after getting frustrated with us. But we don't approve their forum post.
8. May 1st, 2016: That someone (about whom we still don't want to talk about) exposes the vulnerability after getting very frustrated with us. This time, it is exposed on Medium. Because we can't do anything about it, we close the flaw before our extended deadline.

You should really stop continuously assuring us of your continued commitment to product quality and security. Remember what I told you about random words?

Just some extra information from Wordfence who found the Vulnerability

https://www.wordfence.com/blog/2016/05/freshdesk-vulnerability-red-team-exercise/

Let me tell you upfront that as the CEO of Freshdesk, we do not hide anything from customers. We are not that kind of a company.

As you know, we were recently made aware of a security vulnerability in our earlier SSO implementation, that had the potential to impact a small segment of our customers. I can assure you that from the moment we became aware of this vulnerability we have done everything we can to resolve the issue and make our systems more secure. Our immediate action was to shut down that point of access to Freshdesk to protect customer data.

The vulnerability was with our simple SSO implementation.The fix we implemented did require our customers make changes on their end to comply with the new SSO implementation. I know this caused disruptions in their businesses, and for this I am truly sorry.

Is our current SSO implementation robust?

The current Freshdesk SSO implementation uses HMAC-MD5 which is considered safe for use. However, we acknowledge that there are better algorithms available and our team is already in the process of implementing a more robust solution. The current fix was rolled out as an emergency fix to make sure that customer information is safe and secure.

We make every effort to build secure software but like any company, we are not perfect. We will continue to work hard to fix any issues that are brought to our attention, continue to test our own technology both internally and with external security professionals to ensure it is as secure as possible, and have our support team available for any questions or concerns that come up.

With this recent issue we recognize that beyond the vulnerability, we had some operational and communication issues in our response time and remediation. We have already addressed internally and are continuing to look for more ways to improve. The safety and security of our customers’ data is critical for us and we will continue to make it a priority.

I understand that you are frustrated, and you have every right to be frustrated. We apologize to you and all customers that were impacted while we resolved the issue. In the meantime, if you have any further questions, please feel free to reach out to me directly.

Dear Girish Mathrubootham, 

Let's say I am ready to believe what you are saying for a second. You look like a fair guy and I would like to hear honest answers from you, the CEO of Freshdesk, for some of the things I've been asking your colleagues over the last week. So far, they haven't managed to give me straightforward responses. I sincerely hope you will not be so roundabout like them. 

Let me break it down for you so that you can tackle these issues one at a time. 

1. Why did your security team not find this major vulnerability for over two years if they conduct regular audits?
   
   
   



2. How come it wasn't identified even during external audits? You say one of them was conducted by Google, did it really happen?
   
   
   



3. When the vulnerability was finally reported by a customer, why did your team take three days to respond to it? Why did they have to reach out to you via LinkedIn?
   
   
   



4. Why were we given only two days to change our implementation initially?



5. Why did your support team give me wrong information about the SSO script? And why did they refuse to acknowledge it over chat?



6. Why did you take more than two days to respond to my forum post? And why did your Product Manager try to convince repeatedly using random words?



7. Why did you refuse to approve my response to the forum post after your initial deadline?



8. Why was the date extension communicated only through an in-app notification and not email?
   



9. How can your Director of DevOps carelessly say that it was just an example when he introduced the flaw? And say that when it is used by everyone?



10. How can all of this be happening with a company like Freshdesk?
    
    
    
    

"Is our current SSO implementation robust?

The current Freshdesk SSO implementation uses HMAC-MD5 which is considered safe for use."

Just wanted to point out that this is directly taken from the RFC link posted by the CEO:

"It is not urgent to stop using MD5 in other ways, such as HMAC-MD5; however, since MD5 must not be used for digital signatures, new protocol designs should not employ HMAC-MD5."

Given that RFC was published 5 years ago (2011), I'd say that deciding to use HMAC-MD5 and using that link to justify the choice were both extremely poor decisions.

Hello Seriously Security,

When dealing with security vulnerabilities, software companies need to carefully measure the risk of publicly discussing details that could put customer data at risk, against the impact of any remediation actions on customers’ businesses. I want to assure you that all the steps we took in terms of our communications in regard to this incident were done sincerely with that balance in mind.

That said, I believe in being transparent with our customers which is why we shared the details of the incident now that it has been addressed. I understand that you feel that our earlier responses in this forum were not detailed enough. So I will try to provide additional clarity below:

Why did your security team not find this major vulnerability for over two years if they conduct regular audits? How come it wasn't identified even during external audits? You say one of them was conducted by Google, did it really happen?

Every software company, including ours, tries to identify and fix vulnerabilities as quickly as possible. We have undergone regular internal and external audits of our software for this purpose. Since this particular vulnerability was a logical flaw, it couldn't be identified by general black box testing. For external audits, we provided test accounts, with regular login credentials. During this process, SSO was not implemented for them and they did not receive additional credentials to test this part of our product. We acknowledge this oversight.

When the vulnerability was finally reported by a customer, why did your team take three days to respond to it? Why did they have to reach out to you via LinkedIn?

I would like to clarify that we didn’t take three days to respond to Wordfence. We received their first email on the 21st of April 2016 at 3:27 AM IST. And our security team started working on it that morning. By the time we also received the message on LinkedIn, our team was already working on the issue. We acknowledged it to them on the morning of the 23rd of April IST.

I assume Wordfence reached out via LinkedIn because of the nature of the vulnerability. And we started working towards resolving the issue, implemented it, performed security testing and released the fix in a matter of three days. We could have acknowledged Wordfence’s email sooner, but we wanted to make sure that the report was valid before doing so.

In the future, we’ll make sure that we acknowledge probable issues as soon as we receive them, before we go on to validate them.

Why were we given only two days to change our implementation initially?

We apologize for the limited notice with respect to making changes to your SSO. We had to weigh a lot of things in terms of risk to customer data and impact to customers’ businesses in making this decision. We wanted our customer information to be safe, and we had our support team on standby to help customers with changing their implementation. Given the circumstances, I believe this was the right thing to do.

Why did your support team give me wrong information about the SSO script? And why did they refuse to acknowledge it over chat?

This one is on us. We did not communicate well, with our teams internally as well as externally. This has been addressed with the respective stakeholders. We will ensure communication is better in the future.

Why did you take more than two days to respond to my forum post? And why did your Product Manager try to convince repeatedly using random words?

While we do have our support team monitoring and participating in the forums, it has been our expectation that critical issues would be reported directly to customer support, which has a more rapid defined response time. Clearly, we need to make that more apparent for customers. We have increased the level of attention that support gives to our forums but we still recommend customers contact support directly for critical issues.

As I explained previously, security vulnerabilities are a challenge for any software company and we need to carefully weigh the risks to our customers business continuity and the security of their data. Our Product Manager provided a measured response that revealed just enough information as we could reveal at that point. Our intention was to never give you false information. And we will work harder to be sure our communication is more clear.

Why was the date extension communicated only through an in-app notification and not email?

The in-app notification was not our only means of communicating the date extension to our customers. In addition to doing that notification, our support team proactively reached out to customers who asked for an extension to let them know that they had until the 3rd of May to change their implementation.

Since this change was made on the client’s side, we had no way to track if this had already been done or not. We did not want to unnecessarily annoy our customers with reminders when we did not know whether they had done it already. If you had asked for an extension, you would have received this email too. But it looks like you did not, so you didn’t receive the email.

Why did you refuse to approve my response to the forum post after your initial deadline?

Again, it is our responsibility to protect our customers’ data. We delayed approving your response to the forum post initially because we did not want the vulnerability to be made public before our customers were able to fix their implementations. We tried explaining this to you over chat, and I personally wrote to you to try to help you understand the situation.

But when you went on to publish your response on @seri0uslysecur1tyfreshdesk-security-issue-what-are-they-hiding-16aac449d015" style="text-decoration: none;">Medium, you forced us to publish the forum post. By doing this, you put our customers at additional risk. Industry standard practice in the technology business is to report security issues privately and allow the vendor time to fix the issue and communicate with their customers, for this reason.

How can your Director of DevOps carelessly say that it was just an example when he introduced the flaw? And say that when it is used by everyone?

I have worked with Kiran for close to 7 years and I can assure you that he is not the kind of professional that anyone can imagine calling irresponsible. This particular SSO code was a sample implementation published in his personal account. It is human to make mistakes and it is true that this sample implementation had an undiscovered vulnerability. But the expectation was customers would use this as a sample and create their own implementations
Some of our customers have used this script as us. But this implementation was on the client’s side and they had full control to write their own script if they didn’t want to use this one when they started using SSO.

How can all of this be happening with a company like Freshdesk?

As I mentioned at the beginning, all technology companies, large and small, hardware or software, cloud and on-premise, have bugs and vulnerabilities and try to identify and fix these issues as quickly as possible. We’ve acknowledged and corrected this issue, and have published a detailed accounting of the incident.

What is most important is that you are able to confidently serve your customers. Though you have chosen to be anonymous, you are a customer. As such, our phone lines are open and our support teams are ready to address your concerns at any time. Sometimes when conversations are had one-on-one, we can understand each other better and resolve things faster. But, if you feel like you need to transition to another provider, I would be disappointed to see you go but we will make your transition as smooth as possible. It goes without saying that we will issue a full refund.