Weimob Reviews the Database Deletion Incident: All Data Recovered, Compensation Plan Made, Management Held Accountable

Seven days after Weimob’s “database deletion incident,” on March 1, Weimob Group (2013.HK) announced that, as of 8 p.m. on March 1, all deleted data had been fully recovered.

Weimob Group stated that, given the very large volume of data, and to ensure data consistency and the online experience, Weimob will conduct a system go-live drill at 2 a.m. on March 2, and the data recovery will officially go live at 9 a.m. on March 3.

Earlier, on February 25, Weimob Group issued an announcement about the system failure, saying that SaaS (Software as a Service) business data had been deliberately sabotaged by an employee, and that it had reported the case to the Shanghai police; the employee has been placed in criminal detention.

Weimob said the suspect is He, a core operations staff member in the R&D center’s operations department. At 18:56 on February 23, He logged into the company’s internal network jump server via a personal VPN, and maliciously sabotaged Weimob’s online production environment due to personal mental and life issues.

The SaaS product is one of Weimob’s two core businesses; its commercial cloud products cover multiple vertical industries such as e-commerce, retail, dining, local life, and hotel travel — for example, helping merchants build e-commerce sales systems through mini-programs and official accounts. In the first half of 2019, the SaaS business brought Weimob 219 million RMB in revenue, a year-on-year increase of 31.1%, and its SaaS product had 70,000 paying merchants.

This core operations staff’s sabotage of Weimob’s production environment and data led merchants to question the security and stability of Weimob’s system. Over the past 5 trading days, Weimob’s stock price fell 22.33%. During this period, Weimob’s competitors, including Youzan, “took the opportunity” to poach customers.

In the March 1 announcement, Weimob stated that this incident severely impacted merchants’ operations and the management feels deep self-blame and guilt. To address this, Weimob has prepared a 150 million RMB compensation reserve, of which the company bears 100 million and management bears 50 million. Weimob has drafted both a cash compensation plan and a traffic compensation plan for merchants to choose from.

Weimob also traced the responsibility for the accident, saying that although the incident was caused by “human error,” the company’s management bears inescapable responsibility.

Among them, Sun Taoyong, Chairman of the Board and CEO of Weimob, did not attach high importance to data security; Huang Junwei, Executive Director and CTO, did not strictly follow the company’s internal control management system to implement hierarchical and zonal management of operations staff permissions, lacked holistic and forward-looking design in building and introducing the data security technical system, and did not fully implement the security monitoring system. Fang Tongshu, Executive Director and President of the Smart Business Group, as the person in charge of the SaaS business, did not attach high importance to data security, and did not strictly implement the company’s internal control management system or push the R&D side to strengthen data security management.

In the announced data security assurance plan, Weimob stated it will improve the data security management system (covering permissions, monitoring, and auditing) and strictly implement the authorization and approval system; use Tencent Cloud’s CAM permission system for cloud resource management, strictly implement hierarchical authorization and least-privilege principles, and apply secondary authorization for high-risk actions; build a multi-cloud disaster recovery system with full-backup cold-standby architecture in Beijing, Shanghai, Nanjing, and other regions; and leverage Tencent Cloud Database MySQL’s high-availability and security system to gradually abandon self-built database services and migrate to Tencent Cloud Database (CDB), quickly gaining cross-availability-zone and remote disaster recovery capabilities for the database.

Tencent Cloud also announced that the Weimob team has decided to move fully to the cloud and upgrade its data security system through Tencent Cloud’s products and technologies.

Appendix: On March 1, Weimob announced that all data had been fully recovered and published the merchant compensation plan

Dear Weimob merchants:

As of 8 p.m. on March 1, with the assistance of the Tencent Cloud team and after round-the-clock effort, all our data has been fully recovered. Given the very large volume of data, and to ensure data consistency and the online experience, we will conduct a system go-live drill at 2 a.m. on March 2, and the data recovery will officially go live at 9 a.m. on March 3.

This incident severely impacted your operations, and the management feels deep self-blame and guilt. We have prepared a 150 million RMB compensation reserve, of which the company bears 100 million and management bears 50 million. While urgently pushing data recovery, we are also studying the merchant compensation plan in parallel, and have drafted both a cash compensation plan and a traffic compensation plan for you to choose from.

At the same time, this incident exposed management loopholes in our data security. After the incident, we strengthened internal process control management and invited external data security experts to evaluate the data security assurance plan together, quickly formulating a data security assurance plan to prevent such incidents from happening again.

Course of the Incident

February 23: A company employee maliciously sabotaged the company’s online production environment and data, causing the company’s system services to become unavailable. The suspect has been placed in criminal detention by the Baoshan Branch of the Shanghai Public Security Bureau.

February 25: We urgently restored the online production environment for core businesses; new users were unaffected, and we provided temporary transitional solutions for existing users to ensure merchants could operate normally while data was not yet recovered.

February 28: We restored the online production environment for all businesses, reopened login for existing users, and recovered all data for the microsite product.

As of 8 p.m. on March 1, with the assistance of the Tencent Cloud team and after round-the-clock effort, we have fully recovered the data. Given the very large volume of data, and to ensure data consistency and the online experience, we will conduct a data-recovery go-live drill from 2 a.m. to 8 a.m. on March 2; during this period our system will be out of service, and after the drill the system data will be rolled back to the March 2 state.

From 10 p.m. on March 2 to 9 a.m. on March 3, we will formally perform the data recovery go-live. We will restore data from before February 23, and merge the data of February 23 and March 2, at which point all our data recovery will be complete.

Responsibility for the Accident

Although this incident was caused by “human error,” the company’s management bears inescapable responsibility.

First, Sun Taoyong, Chairman of the Board and CEO, did not attach high importance to data security, did not conduct in-depth evaluation and review of the data security assurance plan, did not hire an external expert advisory team to evaluate and test data security, and did not incorporate data security management into daily management.

Second, Huang Junwei, Executive Director and CTO, as the company’s technical lead, did not attach sufficient importance to data security, did not strictly follow the company’s internal control management system for hierarchical and zonal management of operations staff permissions, lacked holistic and forward-looking design in building and introducing the data security technical system, and did not fully implement the security monitoring system.

Fang Tongshu, Executive Director and President of the Smart Business Group, as the person in charge of the SaaS business, did not attach high importance to data security, and did not strictly implement the company’s internal control management system or push the R&D side to strengthen data security management.

Compensation Plan

This incident severely impacted merchants’ operations, and the management feels deep self-blame and guilt. After the incident, while urgently pushing data recovery, the management also studied the merchant compensation plan in parallel.

First, for this compensation plan, we have prepared a 150 million RMB compensation reserve, of which the company bears 100 million and management bears 50 million. Of that, Chairman and CEO Sun Taoyong bears 35 million, Executive Director and CTO Huang Junwei bears 5 million, Executive Director and President of the Smart Business Group Fang Tongshu bears 5 million, and Executive Director and President of the Smart Marketing Group You Fengchun bears 5 million.

Second, in the whole compensation plan, we consider both the profit loss merchants suffered due to system unavailability and the traffic loss caused by the unavailability. Therefore, our compensation plan offers two different options for merchants to choose one.

01 Cash Compensation Plan

We will compensate based on the merchant’s marginal contribution profit during the period of system unavailability, calculated as follows:

Marginal contribution profit = average daily revenue × industry average marginal contribution profit margin × system downtime

(Here, average daily revenue equals the average of the merchant’s actual transaction amount (excluding tax) generated in the Weimob system from 7 p.m. on February 17, 2020 to 7 p.m. on February 23, 2020; the marginal contribution profit margin refers to the ratio of marginal contribution profit to revenue after deducting, from revenue (excluding tax), costs directly related to the sale and delivery of goods and services, such as product cost, warehousing and logistics fees, promotion fees, and sales commissions; the final reference value for the industry marginal contribution profit margin will be based on publicly available research institution reports; system downtime runs from 7 p.m. on February 23 to 9 a.m. on March 3.)

02 Traffic Compensation Plan

For merchants affected by the period of system unavailability, we will provide 50,000 ad impressions on Tencent Ads as traffic compensation, offer account operations services, and extend the SaaS service validity by two months.

(Tencent Ads here includes WeChat Moments ads, WeChat Official Account ads, mini-program ads, etc.; impressions refer to the number of times the ad is seen by users; the operations service includes creative planning, material production, delivery execution, data analysis, account optimization, data reporting, and other operational services.)

Finally, all our compensation will be completed through an online compensation system. The company will develop the online compensation system within about a month, after which merchants can log into the Weimob merchant backend and click to apply for compensation.

Data Security Assurance Plan

This incident exposed management loopholes in our data security. After the incident, while conducting internal system self-checks, we invited external data security experts to evaluate the data security assurance plan together. The measures are announced as follows:

Measure 1: Comprehensively strengthen and rectify the data security management mechanism, and enhance operations platform governance

1. Improve the data security management system (covering permissions, monitoring, and auditing) and strictly implement the authorization and approval system;

2. Use Tencent Cloud’s CAM permission system for cloud resource management, strictly implement hierarchical authorization and least-privilege principles, and apply secondary authorization for high-risk actions;

3. Establish a scientific, efficient, and secure network policy with strict isolation of development, testing, and production environments; replace the self-built bastion host with Tencent Cloud’s bastion host for fine-grained permission grading and authorization management, while strictly auditing bastion host operation logs and sending security audit reports;

4. Strengthen learning of operations security processes, professional ethics, and law.

Measure 2: Strengthen the disaster recovery system, achieving multi-cloud remote cold backup

1. Build a multi-cloud disaster recovery system with full-backup cold-standby architecture in Beijing, Shanghai, Nanjing, and other regions;

2. Leverage Tencent Cloud’s IaaS underlying service capabilities to build a high-availability active-active architecture within the same city;

3. Enable a daily snapshot policy for all cloud hosts to ensure full and incremental backups;

4. Store all unstructured data in Tencent COS object storage for archival, enable COS’s multi-region replication so data resides in multiple locations, and use COS cold storage to ensure data only grows and never shrinks;

5. Establish monthly and quarterly regular drill mechanisms and policies.

Measure 3: Fully move infrastructure to the cloud

1. Leverage Tencent Cloud Database MySQL’s high-availability and security system to gradually abandon self-built database services and migrate to Tencent Cloud Database (CDB), quickly gaining cross-availability-zone and remote disaster recovery capabilities for the database;

2. Fully upgrade Blackstone 1.0 physical machines to Blackstone 2.0 and fully adopt cloud hosts.

Acknowledgements

This incident caused serious adverse effects on merchants, and we sincerely apologize. At the same time, we want to thank the merchants, service providers, partners, investors, and all friends who care about Weimob who still chose to trust us in our darkest hour, and especially the Tencent Cloud team!

The Weimob Team