Algorithm to Find Duplicate Files in System using Hash Map

Given a list of directory info including directory path, and all the files with contents in this directory, you need to find out all the groups of duplicate files in the file system in terms of their paths.

A group of duplicate files consists of at least two files that have exactly the same content.

A single directory info string in the input list has the following format:

“root/d1/d2/…/dm f1.txt(f1_content) f2.txt(f2_content) … fn.txt(fn_content)”

It means there are n files (f1.txt, f2.txt … fn.txt with content f1_content, f2_content … fn_content, respectively) in directory root/d1/d2/…/dm. Note that n >= 1 and m >= 0. If m = 0, it means the directory is just the root directory.

The output is a list of group of duplicate file paths. For each group, it contains all the file paths of the files that have the same content. A file path is a string that has the following format:

“directory_path/file_name.txt”

Example 1:

Input:
[“root/a 1.txt(abcd) 2.txt(efgh)”, “root/c 3.txt(abcd)”, “root/c/d 4.txt(efgh)”, “root 4.txt(efgh)”]
Output:
[[“root/a/2.txt”,”root/c/d/4.txt”,”root/4.txt”],[“root/a/1.txt”,”root/c/3.txt”]]

Note:

• No order is required for the final output.
• You may assume the directory name, file name and file content only has letters and digits, and the length of file content is in the range of [1,50].
• The number of files given is in the range of [1,20000].
• You may assume no files or directories share the same name in the same directory.
• You may assume each given directory info represents a unique directory. Directory path and file info are separated by a single blank space.

Follow-up beyond contest:

• Imagine you are given a real file system, how will you search files? DFS or BFS?
• If the file content is very large (GB level), how will you modify your solution?
• If you can only read the file by 1kb each time, how will you modify your solution?
• What is the time complexity of your modified solution? What is the most time-consuming part and memory consuming part of it? How to optimize?
• How to make sure the duplicated files you find are not false positive?

The files can be stored in a hash map where the keys are the contents, and the values will be a list (array) of file paths. If the content is large, we can store the MD5 hashes (or other hashing algorithms).

Then, we need to get a list of the file paths for each unique content, output only if the list of files is more than 1 (called duplicate files).

The Javascript implementation is given below. We use let files = {} to declare a hash map. Alternatively, we can use let files = new Map() (and use its get() and set() methods).

The Array.map() iterates an array so you won’t need to write the native loop in JS. The string.split() is useful to separate the root directory and a list of files. And the indexOf finds the first location of the target character. Finally, the string.substring() returns a substring that starts at a index.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

/**
 * @param {string[]} paths
 * @return {string[][]}
 */
var findDuplicate = function(paths) {
    let files = { };
    paths.map(function(p) {
        const arr = p.split(" ");
        const dir = arr[0];
        for (let i = 1; i < arr.length; ++ i) {
            let idx = arr[i].indexOf('(');
            const file = dir + "/" + arr[i].substring(0, idx);
            const content = arr[i].substring(idx);
            if (Array.isArray(files[content])) {
                files[content].push(file);
            } else {
                files[content] = [file];
            }
        }
    });
    let result = [];
    Object.values(files).map(function(x) {
        if (x.length > 1) {
            result.push(x);
        }
    });
    return result;
};

/**
 * @param {string[]} paths
 * @return {string[][]}
 */
var findDuplicate = function(paths) {
    let files = { };
    paths.map(function(p) {
        const arr = p.split(" ");
        const dir = arr[0];
        for (let i = 1; i < arr.length; ++ i) {
            let idx = arr[i].indexOf('(');
            const file = dir + "/" + arr[i].substring(0, idx);
            const content = arr[i].substring(idx);
            if (Array.isArray(files[content])) {
                files[content].push(file);
            } else {
                files[content] = [file];
            }
        }
    });
    let result = [];
    Object.values(files).map(function(x) {
        if (x.length > 1) {
            result.push(x);
        }
    });
    return result;
};

We can check if a variable is an Array using Array.isArray() method. When the keys do not appear in the hash map, we need to push an array of 1 element – which contains the file path.

–EOF (The Ultimate Computing & Technology Blog) —